Stream bulk COPYs on sibling connections and coalesce persist backlogs - #707
Closed
aditya1702 wants to merge 4 commits into
Closed
Stream bulk COPYs on sibling connections and coalesce persist backlogs#707aditya1702 wants to merge 4 commits into
aditya1702 wants to merge 4 commits into
Conversation
…with coordinated late commits The live persist ran its three COPY families — transactions(+accounts), operations(+accounts), state_changes — sequentially inside one transaction on one connection, so the insert phase cost their sum and a single Postgres backend did all the index maintenance. They now stream concurrently on three sibling connections, each in its own transaction, while the coordinating transaction stages everything else (assets, contracts, classification, protocol state, token changes, cursor). The table sets are disjoint with no FKs among them, and every goroutine only reads the quiescent buffer. Commits are the visibility point and are held until all four transactions have done their work: siblings commit first (sub-ms each), the coordinating transaction — whose cursor is the authority on which ledgers exist — strictly last. A failure before the first commit rolls everything back and stays retryable exactly as before; a failure after it wraps the new ErrPartialPersist sentinel, which isPermanentPersistError classifies as fatal, because COPY has no ON CONFLICT and re-running the ledger would collide on primary keys. The only crash state this ordering can produce is orphaned bulk rows for the single ledger past the committed cursor, so startup runs IngestStoreModel.DeleteRowsAboveLedger before resuming: one transaction of TOID-bounded deletes (rows of ledgers > cursor are exactly rows with TOID >= toid.New(cursor+1,0,0)) across the five bulk tables, kept chunk-local by each table's chunk skipping. Backfill keeps the single-transaction insertIntoDB path unchanged.
…ommits When the process stage finishes ledgers faster than persist drains them, the persist stage now folds up to --live-persist-max-batch-size (default 5) consecutive ledgers into one commit set: each sibling COPY streams the whole batch on its connection, the coordinating transaction stages the batch's ledgers in order, and the commit barrier fires once — amortizing COPY setup, index-page churn, and fsyncs across the backlog. While the pipeline keeps pace every batch has size 1 and behavior is unchanged. A ledger with classification inputs always opens its own batch: its plan's pool reads see exactly what the previous batch committed, which preserves the deployed-contract-sees-prior-wasm invariant. The cursor stays the authority — one guarded update lands on the batch's last ledger, and a pre-commit failure rolls back and retries the whole batch. wallet_ingestion_persist_batch_size observes coalescing; per-ledger duration histograms record each ledger's amortized share so panels stay comparable across batch sizes.
…on WAL flush SET LOCAL synchronous_commit = off on each sibling session removes up to three serialized fsync waits per persist. Durability is unchanged: the coordinating transaction commits synchronously and strictly last, and its flush covers all earlier WAL including the sibling commit records, so a durable cursor implies durable siblings; rows a crash could lose are exactly the unacknowledged ones startup reconciliation deletes.
The classification-safety cut isolated any ledger carrying protocol wasm/contract observations, but a plan's pool reads are only unsound for inputs no committed batch has classified yet — and re-observations of known contracts are the overwhelmingly common case (synthetic loadtest traffic re-observes the same token contracts every ledger, which cut every batch to size one and disabled batching entirely; goroutine dumps showed the process stage starved of buffers held by the always-cut pending queue). The persist goroutine now keeps seen-sets of wasm hashes and contract IDs folded in after each successful commit; only a ledger introducing unseen inputs opens its own batch. A rolled-back batch marks nothing, and a restart starts empty — conservative until re-warmed.
Contributor
Author
|
Reopened as #711 — the branch was renamed to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Start with: the doc comment on
persistLedgerDataininternal/services/ingest_live.go. Its commit-ordering rules are the contract for everything here.1. Bulk COPYs stream on sibling connections
The three COPY families ran sequentially in one transaction on one connection, so the insert phase cost their sum and one Postgres backend did all the index maintenance.
They now stream concurrently on three sibling connections, each in its own transaction, while a coordinating transaction stages everything else. Table sets are disjoint with no FKs between them.
Commits are the visibility point. Siblings commit first, the coordinating transaction — whose cursor decides which ledgers exist — strictly last.
ErrPartialPersist, fatal. COPY has noON CONFLICT, so a retry would collide on PKs.The only crash state this produces is orphaned rows above the cursor, which
DeleteRowsAboveLedgerclears at startup.2. Backlogs coalesce into batched commits
When process outruns persist, up to
--live-persist-max-batch-sizeconsecutive ledgers fold into one commit set. At normal cadence every batch is size 1 and nothing changes.3 and 4
Siblings commit with
synchronous_commit = off(durability unchanged — the coordinator's synchronous commit covers their WAL). And batches only cut on unseen classification inputs, not any observation — the old rule cut every batch to size 1 and disabled batching entirely.No performance numbers yet — review on design and correctness.
4 of 6 splitting #684. schema → fixes → pipeline → seams → write families → hardening