v0.6.31: the loader records the source byte digest (#118) - #120
Merged
Conversation
Downstream systems pin the file bytes they loaded, but once triples are in the store nothing can recompute file bytes -- the pin was recorded, pattern-checked, and consulted by nothing. A controlled test downstream showed a module composing under a digest of sixty-four zeros. The loader is the only party that ever sees the input bytes. From this change, every load through the turtle funnel (parse_/load_turtle family) records sha256 of the exact bytes the parser consumed: a HashingReader wraps the reader at ingest_dispatch -- the same chokepoint as the #107 lock hook -- so there is no second read and no gap between what was hashed and what was parsed. Recorded on _pgrdf_graphs: source_sha256 digest of the most recent load's bytes source_loads recorded-load count; >1 self-reports that whole-graph byte identity no longer holds NULL in both never recorded, distinct from every digest TDD: all four contract tests watched failing on the missing column before any production code (expected digests computed outside the crate via shasum). One harness lesson paid and documented in the test file: a bare one-statement-per-line fixture sniffs as N-Triples and load_turtle dispatches it to the STAGED loader, whose workers commit own transactions -- inside a #[pg_test]'s wrapping transaction that hangs forever (measured: 31 min at 0% CPU, worker at phase 6 waiting). Fixtures are prefixed Turtle; a follow-up guard is a filed candidate. Not covered yet (stated, not implied): staged loader, v0.6.2 parallel bulk path, TriG/N-Quads quad ingest. Also: local builds now pass PGRDF_BUILD_ID=git describe --tags --dirty through the same build-arg seam CI uses, so a workstation .so names itself instead of reporting unknown. Upgrade path: sql/pgrdf--0.6.30--0.6.31.sql (idempotent ALTERs, the release workflow auto-discovers sql/pgrdf--*--*.sql).
…ot luck The first full-suite run deadlocked: the digest UPDATE ran after the parse, so the loading transaction's LAST act acquired a graphs-table lock while already holding partition-DDL locks from its FIRST act. A concurrent add_graph takes the same pair in the opposite order (graphs INSERT, then partition DDL whose FK locks the graphs table back). Two orders, one cycle, deadlock detected in 1s, three tests down. The record now opens at ingest start (begin_source_record bumps source_loads, taking the graphs-table lock in the same early position add_graph's own INSERT takes it) and closes after the parse (finish_source_record writes the digest under a lock the transaction already holds — nothing is acquired late). Semantics unchanged by construction: a failed parse aborts the transaction and both halves roll back together. Full suite: 373 passed, 0 failed. The deadlock anatomy is documented on begin_source_record where the next reader needs it.
) A source-bytes pin was recorded, pattern-checked, and consulted by nothing — a downstream kernel's controlled test composed a module under a digest of sixty-four zeros. The loader is the only party that ever sees the input bytes; from this release the turtle funnel records their sha256 on _pgrdf_graphs (source_sha256 latest-load-wins, source_loads counts, NULL means never recorded). Release plumbing per the standing pattern: version 0.6.31 across Cargo.toml/lock, pgrdf.control, META.json (both fields), compose mount, CHANGELOG entry; rolling bridge renamed to pgrdf--0.5.1--0.6.31.sql with the column delta appended; regression case 72 extends the _pgrdf_graphs contract with the two nullable columns; 00-smoke expected version strings advance. Closes #118
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.6.31 — the loader records the source byte digest
Closes #118.
Downstream systems pin the file bytes they loaded, but once triples are in the store nothing
can recompute file bytes — the pin was recorded, pattern-checked, and consulted by
nothing (a controlled test downstream composed a module under a digest of sixty-four
zeros). The loader is the only party that ever sees the input bytes. Now it records them.
What lands
_pgrdf_graphs.source_sha256— digest of the most recent load's exact input bytes_pgrdf_graphs.source_loads— recorded-load count;> 1self-reports that whole-graphbyte identity no longer holds
HashingReaderwraps the reader atingest_dispatch— the same chokepoint as the checkpoint lock is enforced at the MCP layer only — SQL clear_graph empties a locked graph, and the lock then blocks the sanctioned repair #107lock hook — so the digest covers the exact bytes the parser consumed: no second read,
no gap between hashing and parsing
sql/pgrdf--0.6.30--0.6.31.sql(idempotent ALTERs; the release workflowauto-discovers
sql/pgrdf--*--*.sql)PGRDF_BUILD_ID=$(git describe --tags --always --dirty)through the same build-arg seam CI uses — a workstation
.sonames itself (dirty flagincluded) instead of reporting
unknownCoverage boundary, stated not implied
The turtle funnel (
parse_/load_turtlefamily) records. The staged loader, the v0.6.2parallel bulk path, and TriG/N-Quads quad ingest do not yet — recorded in the upgrade
script's header and here.
TDD, and the two things the harness taught
All four contract tests were watched failing on the missing column before any production
code existed; expected digests were computed outside the crate with
shasum -a 256.the worker pool preloaded (which the pgrx harness does)
load_turtledispatches it tothe staged loader — whose workers commit their own transactions, inside a
#[pg_test]'s wrapping transaction that can never allow it. 31 minutes at 0% CPU, workerparked at phase 6. Fixtures are prefixed Turtle now; the trap is documented in the test
file; an engine-side guard (staged refuses inside a transaction that cannot commit) is a
candidate follow-up.
transaction's last act acquired a graphs-table lock while already holding partition-DDL
locks from its first act; concurrent
add_graphtakes the same pair in the oppositeorder. Fixed by splitting the record:
begin_source_recordopens at ingest start(graphs-table lock in the canonical early position),
finish_source_recordcloses undera lock the transaction already holds. A failed parse rolls back both halves together.
Verification
-D warningsclean,cargo fmtcleanproposed): the recheck in this PRbody was written before the implementation existed
Merge is not release: the tag cut (version bump, compose mount, CHANGELOG fold) follows the
standing cadence after the merge word.