Problem
tool_args was dropped from SEARCHABLE_COLUMNS (PR #953, LLP 0265 T6) because the indexed tier cannot cover it, and a column the tiers disagree about breaks the invariant the shared allowlist exists to hold: "no tier can surface a match another tier cannot."
The mechanism, for whoever picks this up:
tool_args is { name: 'tool_args', type: 'JSON' } (ai-gateway/src/message_projector.js:84), which lands in parquet as a group with logical_type: {type: 'VARIANT'} and two untyped BYTE_ARRAY children.
searchableStringColumns() in src/core/search/index_worker_thread.js keeps only childless nodes whose converted type is UTF8 or whose logical type is STRING, so tool_args is filtered out and never n-grammed. The same filter exists server-side (hypaware-server/src/search/index-worker-thread.js).
- The indexed read path is two-stage: hypgrep proposes candidate rows from the sidecar, then
rowFilter: accept confirms them (grep_service.js:123). Confirmation already handles tool_args through the matcher's cellText. Only proposal is missing. Any fix that gets those rows proposed closes the gap.
Note this is not a capability the fleet lost. The server declared coverage it never delivered: its row predicate gates on typeof value === 'string' && value !== '' (grep-search.js:452, repeated in toHit at :337), so an object-valued cell has never matched there on any tier, while server LLP 0157 #decision lists tool_args as covered and daemon.js:1174 advertises it in the grep_search tool blurb. Restoring coverage is additive for both repos.
What has to be true first
hypgrep needs to index VARIANT columns, or createIndex needs to accept a value transform / derived text stream so the builder can hand it the rendered JSON. Today createIndex({ sourceFile, sourceMetadata, indexFile, textColumns }) takes parquet column names only. That is the durable fix and it is upstream work, the same shape as widening hypgrep's hyparquet range (see #870).
Alternatives considered and not taken
- Project
tool_args to a real string column at write time. Indexes cleanly, but changes the ai_gateway_messages schema and has structural consumers: skill activation reads tool_args.skill (LLP 0074), program facets read tool_args.command / .cmd (LLP 0075, 0077), graph file resolution (LLP 0023), plus the documented regexp_extract(CAST(tool_args AS VARCHAR), ...) pattern in both hypaware-query SKILL.md files. Needs a migration story for already-written files.
- A companion read on the indexed path for columns the index cannot cover. No schema change, but
tool_args is readKey(block, 'input'), the whole tool input, so Write and Edit calls carry entire file bodies in it. An unbounded per-file decode of that column can cost more than the scan the index was built to avoid. Would need a measured bound first: compare sum(length(cast(tool_args as varchar))) against sum(length(content_text)) on a real cache.
Acceptance criteria, when this is picked up
tool_args returns to SEARCHABLE_COLUMNS only when the indexed tier can propose a row matching solely in it.
- A test proves tier agreement directly: same query, same row, hit from both the scan tier and the sidecar tier after
buildSidecarsForTable.
- The client and the server land it together, or the client lands first and the server picks it up with the shared import; the set is shared and a one-sided change reintroduces the drift.
- Adding coverage neither repo has had is a design change: it needs an LLP extending LLP 0264 #shared (and the server's LLP 0157), not just a code edit.
Design constraints
- The allowlist stays a constant, not configuration (LLP 0264 #shared, LLP 0265 #out-of-scope).
- Until then the coverage clause is accurate as written: only allowlisted columns are searched, and zero hits is not evidence the text is absent elsewhere.
tool_args stays fully readable through hyp query sql.
Do not label this neutral:fix yet - it is blocked on upstream hypgrep, and there is no repair available in this repo.
Problem
tool_argswas dropped fromSEARCHABLE_COLUMNS(PR #953, LLP 0265 T6) because the indexed tier cannot cover it, and a column the tiers disagree about breaks the invariant the shared allowlist exists to hold: "no tier can surface a match another tier cannot."The mechanism, for whoever picks this up:
tool_argsis{ name: 'tool_args', type: 'JSON' }(ai-gateway/src/message_projector.js:84), which lands in parquet as a group withlogical_type: {type: 'VARIANT'}and two untyped BYTE_ARRAY children.searchableStringColumns()insrc/core/search/index_worker_thread.jskeeps only childless nodes whose converted type is UTF8 or whose logical type is STRING, sotool_argsis filtered out and never n-grammed. The same filter exists server-side (hypaware-server/src/search/index-worker-thread.js).rowFilter: acceptconfirms them (grep_service.js:123). Confirmation already handlestool_argsthrough the matcher'scellText. Only proposal is missing. Any fix that gets those rows proposed closes the gap.Note this is not a capability the fleet lost. The server declared coverage it never delivered: its row predicate gates on
typeof value === 'string' && value !== ''(grep-search.js:452, repeated intoHitat:337), so an object-valued cell has never matched there on any tier, while server LLP 0157 #decision liststool_argsas covered anddaemon.js:1174advertises it in thegrep_searchtool blurb. Restoring coverage is additive for both repos.What has to be true first
hypgrepneeds to index VARIANT columns, orcreateIndexneeds to accept a value transform / derived text stream so the builder can hand it the rendered JSON. TodaycreateIndex({ sourceFile, sourceMetadata, indexFile, textColumns })takes parquet column names only. That is the durable fix and it is upstream work, the same shape as widening hypgrep's hyparquet range (see #870).Alternatives considered and not taken
tool_argsto a real string column at write time. Indexes cleanly, but changes theai_gateway_messagesschema and has structural consumers: skill activation readstool_args.skill(LLP 0074), program facets readtool_args.command/.cmd(LLP 0075, 0077), graph file resolution (LLP 0023), plus the documentedregexp_extract(CAST(tool_args AS VARCHAR), ...)pattern in bothhypaware-querySKILL.md files. Needs a migration story for already-written files.tool_argsisreadKey(block, 'input'), the whole tool input, so Write and Edit calls carry entire file bodies in it. An unbounded per-file decode of that column can cost more than the scan the index was built to avoid. Would need a measured bound first: comparesum(length(cast(tool_args as varchar)))againstsum(length(content_text))on a real cache.Acceptance criteria, when this is picked up
tool_argsreturns toSEARCHABLE_COLUMNSonly when the indexed tier can propose a row matching solely in it.buildSidecarsForTable.Design constraints
tool_argsstays fully readable throughhyp query sql.Do not label this
neutral:fixyet - it is blocked on upstream hypgrep, and there is no repair available in this repo.