Skip to content

v0.6.31: the loader records the source byte digest (#118) - #120

Merged
styk-tv merged 3 commits into
mainfrom
feat-118-loader-source-digest
Aug 16, 2026
Merged

v0.6.31: the loader records the source byte digest (#118)#120
styk-tv merged 3 commits into
mainfrom
feat-118-loader-source-digest

Conversation

@styk-tv

@styk-tv styk-tv commented Aug 16, 2026

Copy link
Copy Markdown
Owner

v0.6.31 — the loader records the source byte digest

Closes #118.

Downstream systems pin the file bytes they loaded, but once triples are in the store nothing
can recompute file bytes — the pin was recorded, pattern-checked, and consulted by
nothing
(a controlled test downstream composed a module under a digest of sixty-four
zeros). The loader is the only party that ever sees the input bytes. Now it records them.

What lands

  • _pgrdf_graphs.source_sha256 — digest of the most recent load's exact input bytes
  • _pgrdf_graphs.source_loads — recorded-load count; > 1 self-reports that whole-graph
    byte identity no longer holds
  • NULL in both = never recorded, distinct from every digest; pre-existing graphs read NULL
  • A HashingReader wraps the reader at ingest_dispatch — the same chokepoint as the checkpoint lock is enforced at the MCP layer only — SQL clear_graph empties a locked graph, and the lock then blocks the sanctioned repair #107
    lock hook — so the digest covers the exact bytes the parser consumed: no second read,
    no gap between hashing and parsing
  • Upgrade path sql/pgrdf--0.6.30--0.6.31.sql (idempotent ALTERs; the release workflow
    auto-discovers sql/pgrdf--*--*.sql)
  • Tooling: local builds now pass PGRDF_BUILD_ID=$(git describe --tags --always --dirty)
    through the same build-arg seam CI uses — a workstation .so names itself (dirty flag
    included) instead of reporting unknown

Coverage boundary, stated not implied

The turtle funnel (parse_/load_turtle family) records. The staged loader, the v0.6.2
parallel bulk path, and TriG/N-Quads quad ingest do not yet — recorded in the upgrade
script's header and here.

TDD, and the two things the harness taught

All four contract tests were watched failing on the missing column before any production
code existed; expected digests were computed outside the crate with shasum -a 256.

  1. A hang, measured: a bare one-statement-per-line fixture sniffs as N-Triples, and with
    the worker pool preloaded (which the pgrx harness does) load_turtle dispatches it to
    the staged loader — whose workers commit their own transactions, inside a
    #[pg_test]'s wrapping transaction that can never allow it. 31 minutes at 0% CPU, worker
    parked at phase 6. Fixtures are prefixed Turtle now; the trap is documented in the test
    file; an engine-side guard (staged refuses inside a transaction that cannot commit) is a
    candidate follow-up.
  2. A deadlock, measured: the digest UPDATE originally ran after the parse — the loading
    transaction's last act acquired a graphs-table lock while already holding partition-DDL
    locks from its first act; concurrent add_graph takes the same pair in the opposite
    order. Fixed by splitting the record: begin_source_record opens at ingest start
    (graphs-table lock in the canonical early position), finish_source_record closes under
    a lock the transaction already holds. A failed parse rolls back both halves together.

Verification

  • 4/4 contract tests red on the missing column, then green
  • Full suite 373 passed / 0 failed, clippy -D warnings clean, cargo fmt clean
  • Release modeled before code as a sealed Ticket (state proposed): the recheck in this PR
    body was written before the implementation existed

Merge is not release: the tag cut (version bump, compose mount, CHANGELOG fold) follows the
standing cadence after the merge word.

Downstream systems pin the file bytes they loaded, but once triples are
in the store nothing can recompute file bytes -- the pin was recorded,
pattern-checked, and consulted by nothing. A controlled test downstream
showed a module composing under a digest of sixty-four zeros.

The loader is the only party that ever sees the input bytes. From this
change, every load through the turtle funnel (parse_/load_turtle family)
records sha256 of the exact bytes the parser consumed: a HashingReader
wraps the reader at ingest_dispatch -- the same chokepoint as the #107
lock hook -- so there is no second read and no gap between what was
hashed and what was parsed. Recorded on _pgrdf_graphs:

  source_sha256  digest of the most recent load's bytes
  source_loads   recorded-load count; >1 self-reports that whole-graph
                 byte identity no longer holds
  NULL in both   never recorded, distinct from every digest

TDD: all four contract tests watched failing on the missing column
before any production code (expected digests computed outside the
crate via shasum). One harness lesson paid and documented in the test
file: a bare one-statement-per-line fixture sniffs as N-Triples and
load_turtle dispatches it to the STAGED loader, whose workers commit
own transactions -- inside a #[pg_test]'s wrapping transaction that
hangs forever (measured: 31 min at 0% CPU, worker at phase 6 waiting).
Fixtures are prefixed Turtle; a follow-up guard is a filed candidate.

Not covered yet (stated, not implied): staged loader, v0.6.2 parallel
bulk path, TriG/N-Quads quad ingest.

Also: local builds now pass PGRDF_BUILD_ID=git describe --tags --dirty
through the same build-arg seam CI uses, so a workstation .so names
itself instead of reporting unknown.

Upgrade path: sql/pgrdf--0.6.30--0.6.31.sql (idempotent ALTERs, the
release workflow auto-discovers sql/pgrdf--*--*.sql).
…ot luck

The first full-suite run deadlocked: the digest UPDATE ran after the
parse, so the loading transaction's LAST act acquired a graphs-table
lock while already holding partition-DDL locks from its FIRST act. A
concurrent add_graph takes the same pair in the opposite order (graphs
INSERT, then partition DDL whose FK locks the graphs table back).
Two orders, one cycle, deadlock detected in 1s, three tests down.

The record now opens at ingest start (begin_source_record bumps
source_loads, taking the graphs-table lock in the same early position
add_graph's own INSERT takes it) and closes after the parse
(finish_source_record writes the digest under a lock the transaction
already holds — nothing is acquired late). Semantics unchanged by
construction: a failed parse aborts the transaction and both halves
roll back together.

Full suite: 373 passed, 0 failed. The deadlock anatomy is documented
on begin_source_record where the next reader needs it.
)

A source-bytes pin was recorded, pattern-checked, and consulted by
nothing — a downstream kernel's controlled test composed a module under
a digest of sixty-four zeros. The loader is the only party that ever
sees the input bytes; from this release the turtle funnel records their
sha256 on _pgrdf_graphs (source_sha256 latest-load-wins, source_loads
counts, NULL means never recorded).

Release plumbing per the standing pattern: version 0.6.31 across
Cargo.toml/lock, pgrdf.control, META.json (both fields), compose mount,
CHANGELOG entry; rolling bridge renamed to pgrdf--0.5.1--0.6.31.sql
with the column delta appended; regression case 72 extends the
_pgrdf_graphs contract with the two nullable columns; 00-smoke expected
version strings advance.

Closes #118
@styk-tv
styk-tv marked this pull request as ready for review August 16, 2026 17:55
@styk-tv
styk-tv merged commit b7f9cf5 into main Aug 16, 2026
8 checks passed
@styk-tv
styk-tv deleted the feat-118-loader-source-digest branch August 16, 2026 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Loader records source byte digest — _pgrdf_graphs.source_sha256: make a pinned adoption refusable

1 participant