Skip to content

Restore tool_args grep coverage once hypgrep can index VARIANT columns #977

Description

@bgmcmullen

Problem

tool_args was dropped from SEARCHABLE_COLUMNS (PR #953, LLP 0265 T6) because the indexed tier cannot cover it, and a column the tiers disagree about breaks the invariant the shared allowlist exists to hold: "no tier can surface a match another tier cannot."

The mechanism, for whoever picks this up:

  • tool_args is { name: 'tool_args', type: 'JSON' } (ai-gateway/src/message_projector.js:84), which lands in parquet as a group with logical_type: {type: 'VARIANT'} and two untyped BYTE_ARRAY children.
  • searchableStringColumns() in src/core/search/index_worker_thread.js keeps only childless nodes whose converted type is UTF8 or whose logical type is STRING, so tool_args is filtered out and never n-grammed. The same filter exists server-side (hypaware-server/src/search/index-worker-thread.js).
  • The indexed read path is two-stage: hypgrep proposes candidate rows from the sidecar, then rowFilter: accept confirms them (grep_service.js:123). Confirmation already handles tool_args through the matcher's cellText. Only proposal is missing. Any fix that gets those rows proposed closes the gap.

Note this is not a capability the fleet lost. The server declared coverage it never delivered: its row predicate gates on typeof value === 'string' && value !== '' (grep-search.js:452, repeated in toHit at :337), so an object-valued cell has never matched there on any tier, while server LLP 0157 #decision lists tool_args as covered and daemon.js:1174 advertises it in the grep_search tool blurb. Restoring coverage is additive for both repos.

What has to be true first

hypgrep needs to index VARIANT columns, or createIndex needs to accept a value transform / derived text stream so the builder can hand it the rendered JSON. Today createIndex({ sourceFile, sourceMetadata, indexFile, textColumns }) takes parquet column names only. That is the durable fix and it is upstream work, the same shape as widening hypgrep's hyparquet range (see #870).

Alternatives considered and not taken

  • Project tool_args to a real string column at write time. Indexes cleanly, but changes the ai_gateway_messages schema and has structural consumers: skill activation reads tool_args.skill (LLP 0074), program facets read tool_args.command / .cmd (LLP 0075, 0077), graph file resolution (LLP 0023), plus the documented regexp_extract(CAST(tool_args AS VARCHAR), ...) pattern in both hypaware-query SKILL.md files. Needs a migration story for already-written files.
  • A companion read on the indexed path for columns the index cannot cover. No schema change, but tool_args is readKey(block, 'input'), the whole tool input, so Write and Edit calls carry entire file bodies in it. An unbounded per-file decode of that column can cost more than the scan the index was built to avoid. Would need a measured bound first: compare sum(length(cast(tool_args as varchar))) against sum(length(content_text)) on a real cache.

Acceptance criteria, when this is picked up

  • tool_args returns to SEARCHABLE_COLUMNS only when the indexed tier can propose a row matching solely in it.
  • A test proves tier agreement directly: same query, same row, hit from both the scan tier and the sidecar tier after buildSidecarsForTable.
  • The client and the server land it together, or the client lands first and the server picks it up with the shared import; the set is shared and a one-sided change reintroduces the drift.
  • Adding coverage neither repo has had is a design change: it needs an LLP extending LLP 0264 #shared (and the server's LLP 0157), not just a code edit.

Design constraints

  • The allowlist stays a constant, not configuration (LLP 0264 #shared, LLP 0265 #out-of-scope).
  • Until then the coverage clause is accurate as written: only allowlisted columns are searched, and zero hits is not evidence the text is absent elsewhere. tool_args stays fully readable through hyp query sql.

Do not label this neutral:fix yet - it is blocked on upstream hypgrep, and there is no repair available in this repo.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions