Surfaces and proof: status coverage, the skill learns grep, and the roundtrip smoke (LLP 0265 T7) - #954
Surfaces and proof: status coverage, the skill learns grep, and the roundtrip smoke (LLP 0265 T7)#954bgmcmullen wants to merge 10 commits into
Conversation
… follows (LLP 0265 T6) Compaction is the moment a data file stops changing, so it is the one point where a hypgrep index can be built once and stay valid against its rows. maintainCache now follows every committed rewrite of the grep dataset with a sidecar-build pass over the new generation's files, in a worker thread (createIndex is seconds of straight-line CPU and the daemon is single-threaded), one file at a time. The worker handle and thread are ports of the server's index-worker pair, with one behavioral fix: the worker holds an event-loop ref exactly while a build is in flight, because an always-unref'd worker deadlocks any process whose loop would otherwise drain while awaiting the build. Sidecar existence is the completion marker, no ledger: the publish is a write-then-rename, a killed daemon leaves nothing half-claimed, and the next pass rebuilds whatever is missing. A file whose build keeps failing is quarantined after three attempts (in-memory, process-lifetime; a restart is the retry) and the scan tier serves it forever after: index presence is purely a performance property. The build pass can never fail the partition's own maintenance verdict. Two hazards found and closed on the way: - countDataFiles and measureDataDir counted sidecars (*.parquet in data/), which would have made every just-indexed partition read as "grew since compaction" and rewrite itself every tick through the LLP 0199 baseline gate. Both now exclude .index.parquet; a test pins that a second unforced tick stays converged. - A corrupt sidecar used to fail the whole search; the indexed tier now runs into local buffers and commits only on success, so an unreadable sidecar degrades that one file to the brute scan with no double count. GREP_DATASET joins the shared searchable-columns module so the search service and the build pass cannot disagree about which dataset carries indexes. Tests: per-file build and existence-marker idempotency, the quarantine budget with the scan tier still serving, the corrupt-sidecar fallback, maintenance building indexes for exactly the grep dataset, sidecars not re-triggering compaction, and a retired generation dying whole with its sidecars inside (the no-GC-code guarantee). The compaction-effectiveness tests' liveDataFiles helper learns the same sidecar exclusion the production counters did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sion-safe publish, and failures that name their file - Hoist `sidecarPathFor` beside `GREP_DATASET` in searchable_columns.js. The build pass and the search service each carried their own copy of the `.index.parquet` rule; two copies of a path contract drift into a build that writes an index nobody probes for. - Give the publish scratch file a random token. A fixed `<sidecar>.tmp` is only atomic for a single writer: the daemon tick and a hand-run `hyp` over the same cache would interleave into one scratch file and rename the mixture into place as a finished sidecar. The scratch file is now also removed on the failure path. - Name the data file on `grep_index.build_failed` / `grep_index.file_quarantined` / `grep_search.sidecar_unreadable`, and add the component/operation attributes, so three warnings can be told apart as one poisoned file or three. - Append rather than spread the indexed tier's buffered hits: `limit` reaches the service unvalidated and one file can fill the budget. - Correct the module docs: the pass runs only behind a committed compaction, which always publishes a fresh generation, so it never re-attempts a file it skipped or failed on, and a daemon restart is not a retry. Add the LLP 0264#lifecycle ref the module realizes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he sidecar name Findings from the neutral review of #954, all low severity, no behavior change to the shipped surfaces: - The roundtrip smoke's "newest visible hit leads" check accepted either sess-purged or sess-new, so a one-day sort inversion passed it. The seed dates make the answer exact; pin it to sess-new. - The smoke's header claimed step 3 runs `hyp query maintain --force`, but the step calls `maintainCache` directly (it asserts on the sidecar counters in the returned report). Say what the code does. - The post-shutdown telemetry assertions were wrapped in `step()`, which opens a root span against an already-shut-down provider: the smoke_step never reached the trace. Unwrapped, matching every sibling flow. - `countIndexedDataFiles` restated the `<file>.index.parquet` pairing rule that `sidecarPathFor` already owns, in the one place a drift would silently misreport coverage. Import it instead. - `search-sidecar-build.test.js` claimed indexedFileCount "stays absent elsewhere" without a partition where it could be absent. Add one. - The hypaware-query skill pointed at `hyp query status` two paragraphs after routing cache operations to `hyp cache`. Both names work (alias); use the canonical one. The divergence fixture hashes host-only lines, so an identical edit to both copies leaves it unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neutral review - PR #954 @
|
neutral review - PR #954 @
|
neutral is stuck on PR #954 and needs a human decisionWhat neutral was doing. The triage rung on PR #954 ( Why it cannot proceed. Round 2 finding 1: this branch and PR #953 each fixed the same duplication in opposite directions, and the tree breaks the instant they meet.
Triage reproduced the failure rather than inferring it: merging That is a load failure of It cannot be fixed on this branch alone: this branch's import { buildSidecarsForTable } from '../search/sidecar_build.js'
import { GREP_DATASET, sidecarPathFor } from '../search/searchable_columns.js'Non-blocking residuals: none. Every other finding from both rounds is verified landed at head What neutral needs from you. A decision on where the one-line integration fix is applied, since it spans two open PRs and merge order is not triage's to drive. Options:
How to unstick. Reply with a comment on this PR (or push to the branch); neutral monitors this thread and will re-engage with your guidance on its next tick. |
Unstick: option 1 from your triage comment. #953 lands first.Keep #953's move of So: #953 merges first (its One thing to re-check while you are in there: #953's answer removes |
|
Re-engaging with your guidance - option 1 taken: #953 lands first keeping the |
…: nine columns, tool_args moves to the not-searched list The maintainer's unstick on PR #954 settled that #953 drops tool_args from SEARCHABLE_COLUMNS. The verb's coverage clause interpolates the constant so it follows on its own, and the query_grep_roundtrip smoke never seeds or asserts a tool_args hit, so the SKILL.md enumeration (both host copies, one shared line) was the only surface stating the old set. The two deliberate host-only lines are untouched, so the divergence fixture needs no re-record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neutral triage - PR #954 @
|
…lowlist Merge of the base branch plus the maintainer's decision on #953. ## Conflict: src/core/search/grep_service.js Both branches independently fixed "a poisoned sidecar fails the whole query", so the conflict is two spellings of one fix, not two intents. - t4 (`b590b110`) hoisted a `searchIndexed` helper that collects through the shared `collect`/`trimHits` machinery (its `8b694e5b` sort-order truncation) and degrades ONLY when the read failed before the first row; a later failure propagates, because the rows already pushed to the shared buffer could not be taken back. - t6 buffered the attempt into a local array and committed on success, so any failure degrades, and logged `grep_search.sidecar_unreadable` naming the file that needs deleting. Taken: t4's structure with t6's buffering and log folded in. The helper keeps its name, the 2-argument `isAbort(err, signal)` t4 introduced (t6 still called the 1-argument form, which would have turned a timeout into a thrown error), the day-descending early break, and sort-order truncation; it gains the local buffer, so a sidecar that tears mid-read degrades that one file instead of failing the query, and it gains the warning. `trimHits` is generalized to `trimBuffer(list)` so the per-file buffer is trimmed by the same rule as the shared one: buffering must not trade the walk's memory bound away, and cutting the buffer in walk order would reintroduce the bug `8b694e5b` fixed. ## Decision on #953: tool_args leaves SEARCHABLE_COLUMNS `tool_args` is the dataset's one VARIANT column. The index worker only indexes STRING leaves in the allowlist and the server's row predicate gates on `typeof value === 'string'`, so the column has never produced a hit on any tier in either repository: T3's `cellText` coercion made the client's scan tier uniquely able to match it, on a premise ("the indexed tier reads the column's own text") that was false on both sides. Rather than add coverage neither repo has ever had, the column is dropped and the loss is recorded. Follow-up: #977. - `SEARCHABLE_COLUMNS` loses `'tool_args'`; `SCAN_COLUMNS` derives from it, so the brute scan stops decoding the column too. - The module comment now gives the VARIANT reason and points at #977, in the spirit of server LLP 0157 #identifier-columns. - `cellText` keeps its coercion (it is what keeps `rowTest`, `test` and `locate` answering identically on any cell shape, and #977 needs it in place), but its comment stops claiming the indexed tier reads the column's text. Its `@ref` gloss is corrected to match. - `toHit` and `GrepSearchMatcher` lose the same false claim. - Tests: the pinned allowlist drops the column, a new test pins its absence from both the allowlist and the scan projection, and the matcher tests exercise the coercion through a column that is actually searchable while pinning that `tool_args` no longer matches. - New in `search-grep-service.test.js`: a row matching only in `tool_args` returns zero hits from BOTH tiers, scan and indexed, with the tier counters proving each one really served the file. The invariant is tier agreement, not coverage. No LLP change: LLP 0264 never enumerates the columns (it points at this module and defers to server LLP 0157), so nothing it settled is contradicted. Amending server LLP 0157 and the now-wrong `grep_search` tool blurb is hypaware-server work, tracked separately. `sidecarPathFor` stays the single contract owner in `searchable_columns.js`; no re-export from `sidecar_build.js` is restored (#954 takes the import fix on its side). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… scratch is not data bytes The merge resolution's per-file buffer degraded one file instead of failing the query, but it also made a deadline throw away everything the index had already produced for the file it landed in. hypgrep checks the signal at every coalesced range boundary, so a deadline lands inside a file, and on a newest-first walk that is the newest file the caller most wants. Committed before the abort propagates: safe because an abort ends the walk, so the file is never rescanned and no row can be counted twice. grep_search.sidecar_unreadable named only the sidecar, but parquetFind opens the source data file through the same factory and runs the row filter per row, so a torn source parquet lands in that catch too and points the operator at a healthy index. Renamed to grep_search.indexed_read_failed and both files are named. measureDataDir excluded `*.index.parquet` but not the build's publish scratch, `<file>.index.parquet.<uuid>.tmp`, which survives a kill between write and rename with no reaper until the generation retires. countDataFiles already skips it, so counting its bytes broke the shared-file-set invariant in the dangerous direction: needsCompaction compacts on a LOW average, so a large orphan makes a fragmented partition read as healthy. Test pins it. Also recorded, not fixed: a sidecar freezes the allowlist it was built over (hypgrep stores hypgrep.text_columns in the index and prunes to it, and nothing compares that stamp to today's SEARCHABLE_COLUMNS), so #977 has to invalidate existing sidecars rather than only build new ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b25c9cf to
260e9b4
Compare
…oundtrip smoke (LLP 0265 T7) The last task of the grep plan, on top of T5 (the verb) and T6 (the sidecar build): - hyp query status reports grep-index coverage: a summary line (grep index: N of M data files indexed) plus an indexed= extra on each searchable partition, computed by a pure directory scan in cacheStatus (indexedFileCount, grep dataset only). "Grep is slow on deep history" is now diagnosable where the operator already looks, and the expected gap (fresh files index only at compaction) is explained in the line itself. - The hypaware-query SKILL.md (both host copies) documents the grep subcommand: when to prefer it over LIKE-SQL, the ten-column coverage caveat and what zero hits does not prove, the truncation notice, the coverage-versus-speed relationship with the status line to check, the sub-ngram literal cliff (a short literal defeats index pruning but never correctness), and the local-only withholding parity with SQL. The read-class verb lists gain grep_search. The host-divergence fixture is re-recorded (the two deliberately host-specific lines both carry the verb list). - A hermetic smoke, query_grep_roundtrip, drives the real CLI through the whole story: scan-tier search before any index exists, hyp purge --session removing a row grep can then no longer surface, hyp query maintain building sidecars, the status coverage line, the indexed tier answering identically (proved from query.grep_search span attributes: indexed>0 scanned=0, and no query text in any span), and LLP 0105 withheld/visible/override from three caller contexts. Added to the release battery in AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he sidecar name Findings from the neutral review of #954, all low severity, no behavior change to the shipped surfaces: - The roundtrip smoke's "newest visible hit leads" check accepted either sess-purged or sess-new, so a one-day sort inversion passed it. The seed dates make the answer exact; pin it to sess-new. - The smoke's header claimed step 3 runs `hyp query maintain --force`, but the step calls `maintainCache` directly (it asserts on the sidecar counters in the returned report). Say what the code does. - The post-shutdown telemetry assertions were wrapped in `step()`, which opens a root span against an already-shut-down provider: the smoke_step never reached the trace. Unwrapped, matching every sibling flow. - `countIndexedDataFiles` restated the `<file>.index.parquet` pairing rule that `sidecarPathFor` already owns, in the one place a drift would silently misreport coverage. Import it instead. - `search-sidecar-build.test.js` claimed indexedFileCount "stays absent elsewhere" without a partition where it could be absent. Add one. - The hypaware-query skill pointed at `hyp query status` two paragraphs after routing cache operations to `hyp cache`. Both names work (alias); use the canonical one. The divergence fixture hashes host-only lines, so an identical edit to both copies leaves it unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…h completeness notices, and the two remote exceptions The doc is the surface an agent acts from, so a caveat it omits becomes a wrong call, not just a gap: - `--limit` was named as the remedy for truncation with no ceiling. The verb accepts any integer through its schema and then silently substitutes the default for anything outside 1-1000, so `--limit 5000` returns 50, fewer than the caller asked for and with no signal. Stated. - Only one of the verb's two stderr completeness notices was documented. `grep: the search stopped before covering every file` means the walk aborted, which a wider limit cannot fix; reading it as truncation sends a caller the wrong way. Both are now named and told apart. - `--remote` was described as running "the same search", but a server restricts `--regex` to its operator and rejects `--include-local-only` outright. Both exceptions stated beside the flag. Also aligns the new quick-reference line's trailing comment with its neighbours (it sat one column right). Both host copies carry identical edits; all four touched lines are shared, so the host-divergence fixture is unchanged.
…: nine columns, tool_args moves to the not-searched list The maintainer's unstick on PR #954 settled that #953 drops tool_args from SEARCHABLE_COLUMNS. The verb's coverage clause interpolates the constant so it follows on its own, and the query_grep_roundtrip smoke never seeds or asserts a tool_args hit, so the SKILL.md enumeration (both host copies, one shared line) was the only surface stating the old set. The two deliberate host-only lines are untouched, so the divergence fixture needs no re-record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The T7 review commit that stopped restating the sidecar name imported sidecarPathFor from sidecar_build.js, which is where it lived when T7 was branched. #953's own review round moved it beside GREP_DATASET in searchable_columns.js, so re-stacking T7 on the current T6 tip left the import naming an export that is no longer there: a typecheck error and every maintenance test red. Point it at the module that exports it now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
891f912 to
e45791b
Compare
neutral triage - PR #954 @
|
|
Superseded by #984, which carries this work (and every review fix from this PR) on a single branch cut from current master. The stack could not be merged in sequence: this repo squash-merges, so merging one PR put a commit on master sharing no history with the branches behind it, and each remaining PR then re-proposed its parent's changes. Verified by simulation: merging #951 and then #953 produced add/add conflicts in Integrating on current master also surfaced two problems invisible here: a The branch is retained, so this can be reopened if needed. This PR's task: T7, surfaces and the roundtrip smoke. |
What this is
LLP 0265 T7, the final task. It needs both T5 (#952) and T6 (#953), so this branch is T5 with T6 merged in; the PR is based on the T5 branch and the first few commits shown are T6's. Merge order: #951 (T4) -> #952 (T5) and #953 (T6) in either order -> this.
The three surfaces
hyp query statusreports index coverage. A summary line -grep index: N of M data files indexed (searches brute-scan the rest; compaction indexes them)- plus anindexed=extra on each searchable partition.cacheStatuscomputes it with a pure directory scan (indexedFileCount, grep dataset only), matching the cost profile of the other status counters. "Grep is slow on deep history" becomes diagnosable where the operator already looks, and the expected gap is explained in the line itself.hypaware-queryskill learns grep, in both host copies: a## Full-text searchsection covering when to prefer it overLIKE '%...%'SQL, the ten-column coverage caveat (fixing the "core query subcommands are overview, schema, and sql" claim, and spelling out what zero hits does not prove), the truncation notice, the coverage/speed relationship with the status line to check, the sub-ngramLengthliteral cliff (a short literal defeats index pruning, never correctness), and local-only withholding parity with SQL. The read-class verb lists gaingrep_search. The host-divergence fixture is re-recorded since the two deliberately host-specific lines both carry that list.query_grep_roundtrip(added to the release battery in AGENTS.md), drives the real CLI dispatch end to end:hyp purge --sessionremoving a row grep can then no longer surface,hyp query maintain --forcecompacting and building sidecars,query.grep_searchspan attributes (indexed_file_count>0, scanned_file_count=0), with a guard that no span carries the query text,--include-local-only.Everything the plan asked T7 to assert is asserted; the one variation is that the stale-sidecar purge case (sidecar built before the purge) lives in the T6 unit tests rather than the smoke, where the natural flow builds the sidecar after.
Verification
npm run smoke -- query_grep_roundtrip: ok, first run.npm run typecheck: clean; unit suites for sidecar-build (now also pinningcacheStatus.indexedFileCount) and the verb: green.npm test: green modulo the two known environment-only failures noted on The local grep service: one newest-first file walk, sidecar-or-scan per file (LLP 0265 T4) #951.With this, the plan's task graph is complete: T1-T3 merged, T4-T7 open as this stack. What remains outside the plan is server-repo work only (the import swap onto
hypaware/core/searchafter the dependency bump).🤖 Generated with Claude Code
neutral triage @
e45791b7Verdict: zero residual findings; nothing left to defer. The one finding open after two review rounds (round 2 finding 1, the
sidecarPathForimport that would break the tree when #953 merged in, tracked as #979) is applied at this head:maintenance.js:21-22now importssidecarPathForfrom../search/searchable_columns.js, which exports it, and #953's content (including thetool_argsallowlist drop) is already in this branch's history. The branch was rebased since the previous triage, so all eleven findings from both rounds were re-verified against this tree file by file; all are landed. The maintainer's re-check items hold: the SKILL.md coverage bullet names exactly the nine post-#953 columns in both host copies, the verb coverage clause interpolatesSEARCHABLE_COLUMNS, and thequery_grep_roundtripsmoke referencestool_argsnowhere.Verified on this head, per the unstick request:
npm run typecheckclean;npm test4650 pass, 0 fail;npm run smoke -- query_grep_roundtripok; CI green. The PR waits only on stack merge order (#951, then #952/#953 in either order, then this). #979 stays open until this PR merges, since the fix it tracks lives on this branch.