You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
build_merge silently drops nodes from files entirely outside the --update's scope (no ID collision, no warning) — shrink guard catches it, root cause undiagnosed #3477
An ordinary incremental graphify --update on a mature, actively-maintained graph produced a merge candidate missing 313 nodes belonging to 313 files that were not part of this run's changeset at all — not re-extracted, not pruned, not touched in any way. None of the 313 dropped IDs collide with any ID the new extraction produced. The #479 shrink guard correctly caught the resulting net loss and refused to overwrite graph.json, so no data was lost on disk — but the mechanism producing the loss during the merge itself is undiagnosed, and this is not a one-off: the corpus was previously rebuilt from scratch specifically to clear a node-loss problem, and the same class of symptom has now recurred during normal incremental operation. A full rebuild is not a durable workaround for whatever this is.
graphify --update via the standard skill path, dedup=True (default)
Corpus: 673 files, ~3 months of incremental history, graph at 5,262 nodes prior to this run
What happened
Run scope: 55 files changed since the last update (49 doc + 5 paper), 14 already cached, 40 requiring fresh semantic extraction. Split into 2 subagent chunks of 20 files each — both completed cleanly, no JSON validation errors, no dangling refs: 227 new nodes combined, 293 edges, 3 hyperedges per chunk. AST extraction on the 1 changed code file: 6 nodes, 13 edges, clean.
build_merge() on this input produced 4,951 nodes — verified directly against the on-disk files:
A net loss of 311, which correctly tripped the #479 shrink guard — graph.json was refused a write and remains at 5,262 nodes, confirmed untouched (mtime predates this run).
Diagnosis so far
Diffed old-graph node IDs against the new merge candidate: 573 old nodes are absent from the new output.
260 come from the 40 files that were legitimately re-extracted this run — expected, their old records are supposed to be replaced.
313 come from files nothing in this run touched. Checked directly: zero overlap between these 313 IDs and the full set of IDs the new extraction produced (no collision, fuzzy or exact, that would explain a legitimate merge). They are properly path-namespaced per the project's ID convention — e.g. commissions_comm_075_atlas_stage_a_comm_075, commissions_comm_089_b7_characterization_preregistered_branches_0_4b — not the kind of generic/global-style ID (e.g. person_emma) that would plausibly collide across files.
Separately, the same merge logged 5 explicit minted by two different files warnings for genuine shared-entity collisions (person_emma, person_vera, paper_shai_2024_belief_state_geometry, two syntheses_grammar_driven_model_* nodes). Those are correct, expected consolidation, and have zero overlap with the 313 — they are not part of the unexplained loss.
The merge's own log reported Deduplicated 294 node(s), 293 exact + 1 fuzzy for this call. The count is suspiciously close to the scale of the unexplained loss, which is why dedup is the leading suspect — but I have not isolated which specific pass or key is responsible.
Nothing was forced past the guard. On disk and available if useful:
.graphify_extract.json — the full 4,951-node merge candidate
Semantic cache for the 40 re-extracted files (so this can be replayed without re-running the subagent extraction)
The full list of 313 dropped node IDs and the diff methodology, available on request
One operational loose end from stopping mid-pipeline
manifest.json was already stamped for the 40 re-extracted files — that write happens before the guard fires, in save_manifest. A plain graphify --update run right now would likely treat those 40 files as already processed even though nothing from this run actually reached graph.json. Left as-is rather than hand-patched, since the manifest schema wasn't understood well enough to edit safely without risking a different kind of silent corruption.
Impact
Silent node loss with no collision and no log trace, on a merge the shrink guard happens to catch by total count — but a smaller or differently-shaped loss on the same corpus might not cross that threshold and would go completely unnoticed. This is the second time this specific corpus has hit a node-loss problem serious enough to warrant a from-scratch rebuild; the recurrence during ordinary incremental use suggests the underlying cause survived that rebuild.
Summary
An ordinary incremental
graphify --updateon a mature, actively-maintained graph produced a merge candidate missing 313 nodes belonging to 313 files that were not part of this run's changeset at all — not re-extracted, not pruned, not touched in any way. None of the 313 dropped IDs collide with any ID the new extraction produced. The#479shrink guard correctly caught the resulting net loss and refused to overwritegraph.json, so no data was lost on disk — but the mechanism producing the loss during the merge itself is undiagnosed, and this is not a one-off: the corpus was previously rebuilt from scratch specifically to clear a node-loss problem, and the same class of symptom has now recurred during normal incremental operation. A full rebuild is not a durable workaround for whatever this is.Environment
graphifyy0.9.56 (~/.local/share/uv/tools/graphifyy/)graphify --updatevia the standard skill path,dedup=True(default)What happened
Run scope: 55 files changed since the last update (49 doc + 5 paper), 14 already cached, 40 requiring fresh semantic extraction. Split into 2 subagent chunks of 20 files each — both completed cleanly, no JSON validation errors, no dangling refs: 227 new nodes combined, 293 edges, 3 hyperedges per chunk. AST extraction on the 1 changed code file: 6 nodes, 13 edges, clean.
build_merge()on this input produced 4,951 nodes — verified directly against the on-disk files:A net loss of 311, which correctly tripped the
#479shrink guard —graph.jsonwas refused a write and remains at 5,262 nodes, confirmed untouched (mtime predates this run).Diagnosis so far
Diffed old-graph node IDs against the new merge candidate: 573 old nodes are absent from the new output.
commissions_comm_075_atlas_stage_a_comm_075,commissions_comm_089_b7_characterization_preregistered_branches_0_4b— not the kind of generic/global-style ID (e.g.person_emma) that would plausibly collide across files.Separately, the same merge logged 5 explicit
minted by two different fileswarnings for genuine shared-entity collisions (person_emma,person_vera,paper_shai_2024_belief_state_geometry, twosyntheses_grammar_driven_model_*nodes). Those are correct, expected consolidation, and have zero overlap with the 313 — they are not part of the unexplained loss.The merge's own log reported
Deduplicated 294 node(s), 293 exact + 1 fuzzyfor this call. The count is suspiciously close to the scale of the unexplained loss, which is why dedup is the leading suspect — but I have not isolated which specific pass or key is responsible.Related issues checked, ruled out
(basename, label)collision) — fixed in 0.9.23 via full-path keying, confirmed dead as recently as Sep 2; doesn't apply at 0.9.56, and this run has zero ID/label collision with the dropped nodes regardless.build_mergereplace-on-re-extract deletes a source's own nodes before validating the new extraction;#479guard skipped underdedup=True) — mechanistically adjacent (same guard, samededup=Truedefault) but the wrong scope: that issue is about a re-extracted file under-producing on itself. This report is about nodes vanishing from files outside the changeset entirely. Also worth flagging as a data point either way: build_merge: replace-on-re-extract deletes a source's nodes before the new extraction is validated; the #479 guard is skipped under dedup and delegates to a #1951 guard that does not exist in the package #3004 states the#479guard is skipped wheneverdedup=True(the default) — but the guard clearly fired here. Either that skip condition has narrowed since 0.9.48, or this loss pattern doesn't route through the same code path build_merge: replace-on-re-extract deletes a source's nodes before the new extraction is validated; the #479 guard is skipped under dedup and delegates to a #1951 guard that does not exist in the package #3004 describes.#479guard misreads a legitimate dedup win as data loss via total-count accounting) — wrong direction; this is a real loss the guard correctly caught, not a false positive.Preserved for further diagnosis
Nothing was forced past the guard. On disk and available if useful:
.graphify_extract.json— the full 4,951-node merge candidateOne operational loose end from stopping mid-pipeline
manifest.jsonwas already stamped for the 40 re-extracted files — that write happens before the guard fires, insave_manifest. A plaingraphify --updaterun right now would likely treat those 40 files as already processed even though nothing from this run actually reachedgraph.json. Left as-is rather than hand-patched, since the manifest schema wasn't understood well enough to edit safely without risking a different kind of silent corruption.Impact
Silent node loss with no collision and no log trace, on a merge the shrink guard happens to catch by total count — but a smaller or differently-shaped loss on the same corpus might not cross that threshold and would go completely unnoticed. This is the second time this specific corpus has hit a node-loss problem serious enough to warrant a from-scratch rebuild; the recurrence during ordinary incremental use suggests the underlying cause survived that rebuild.