Build grep sidecars at compaction so the indexed read path lights up (LLP 0265 T6)

[philcunliffe]

Problem

LLP 0265 #sequencing makes the scan tier the complete, correct search and the sidecar index pure acceleration behind it, exactly the property server LLP 0130 designed for ("index presence is purely a performance property"). T4 builds the scan tier and already exercises the indexed read path against a hand-built sidecar. Nothing yet builds those sidecars, so the indexed path stays dormant and deep-history greps pay the full walk.

Goal

Build a grep sidecar for each data file that compaction finalizes, so the indexed read path T4 already supports lights up, and pin with a test that sidecars die with the files they index.

Acceptance criteria

• Port the server's build hardening rather than reinventing it. LLP 0265 #verified records it as directly portable: the worker-thread handle (hypaware-server/src/search/index-worker.js, 149 lines), the three-attempt poison bound (server LLP 0158), and sidecar existence as the idempotency marker (server LLP 0128).
• compactGeneration queues an index build for each finalized data file (src/core/cache/maintenance.js:800), over the T3 allowlist columns. Compaction is the cadence; there is no separate scheduler.
• Sidecar existence is the marker. No ledger. A rebuild is a no-op when the sidecar is there; a build that has failed three times is quarantined and not retried forever.
• A build failure never fails compaction and never fails a query. A file with no sidecar, or a quarantined one, is served by the scan tier and returns the same rows, just slower. This is the whole point of the sequencing: an index that fails to build degrades performance, never correctness.
• Sidecars are data/-adjacent, so the existing GC reclaims them. LLP 0265 #verified confirms the orphan sweep and retention already delete recursively (src/core/cache/maintenance.js:1364, src/core/cache/retention.js), so this task writes no new GC and instead pins the existing one with a test.
• Tests in root test/:
  • a finalized data file gets a sidecar over the allowlist columns;
  • the orphan sweep deletes sidecars with their generation;
  • retention deletes sidecars with their files;
  • an unindexed file is served by the scan tier with the same rows;
  • a quarantined file (three failed attempts) is served by the scan tier and is not retried;
  • the build is idempotent: a second compaction pass over an already-indexed file does no work.
• Disk growth from sidecars is bounded and reported in the PR description. Sizing input: T4 (Local grep service: walk the cache newest-day-first and return server-shaped hits (LLP 0265 T4) #926) measures the 90-day narrow-projection walk cost; LLP 0265 #verified lists that measurement as not-yet-verified and says T6 is sized by it. If Local grep service: walk the cache newest-day-first and return server-shaped hits (LLP 0265 T4) #926 has landed its measurement, use it; if not, state the assumption you sized against explicitly in the PR.

Design constraints

• This task is parallel to T4 and T5, not downstream of them: its deps are [T1, T3] only. It must not import from grep_service.js or the verb, and must not require them to exist.
• No config knob for indexed columns or index cadence: the allowlist is the shared constant by decision (LLP 0264 #shared) and the cadence is compaction's. LLP 0265 #out-of-scope rules the knob out explicitly.
• hypgrep needs hyparquet-writer only for createIndex; the root already carries the writer as an optionalDependency on the cache write path, and T1 pins it to 0.16.6 beside the exact hyparquet 1.28.2 floor (LLP 0222 #hyparquet-floor). Do not add a second copy or relax the floor.
• The index build runs off the main thread (the worker handle above): compaction must not block on it, and a wedged build must not wedge maintenance.
• Repo style per CLAUDE.md: JavaScript, no semicolons, no em dashes anywhere, JSDoc types (no TypeScript, no @typedef), interfaces in .d.ts imported via @import, repo-root-anchored .js type-import specifiers.
• Annotate the non-obvious decisions this realizes with @ref LLP NNNN#anchor where a ref tells the reader something the code and filename do not. LLP 0209 is the compaction pass this hooks.

Depends on

T1 (hypgrep dependency and the hyparquet / hyparquet-writer overrides, PR #877) and T3 (src/core/search/ allowlist and shapes, PR #876). Both are held for a human merge; this task cannot land before they do.

---

Wave 4 (deps [T1, T3], parallel to T4/T5), task T6 of LLP 0265 #tasks. Decision: LLP 0264. Complexity 3. Unblocks T7 (with T5).

[philcunliffe]

neutral looked at this and is waiting, not stuck.

Re-derived from origin/master this tick:

• src/core/search/ does not exist on master (git ls-tree origin/master src/core/search/ is empty) - T3 is still open in PR Hoist the shared grep-search modules into src/core/search/ and publish them as ./core/search #876.
• hypgrep appears nowhere in master's package.json - T1 is still open in PR hypgrep enters the dependency set behind an overrides pin that holds the exact LLP 0222 hyparquet floor #877.

Both dep PRs are reviewed and held for a maintainer merge. Until they land there is nothing on master for this task to build on, and neutral will not branch an implementation off another open PR's head (that would race a PR whose head can still move).

No neutral:stuck label, deliberately: the blocker here is a state change, not a decision. Sticking would need a human reply to clear even after the deps merged, which would strand this issue. Instead neutral re-checks the two conditions above on every tick and picks this up automatically on the first tick after #877 and #876 are on master.

Nothing is needed from you except merging those two when you are ready.

[philcunliffe]

Implemented by PR #953 (grep/t6-sidecar-build, authored by @bgmcmullen), which neutral has adopted and is reviewing.

Removing neutral:fix so the maintenance reconciler does not open a competing fix/issue-927 branch that would duplicate that PR. This is a de-duplication of authorization, not a decision that the issue is done: it stays open until #953 merges.

Re-add neutral:fix if you want neutral to work this issue independently of #953.