Rearchitect live ingestion for high-TPL ledgers: pipeline, batched 5-way COPY commits, once-per-op decoding, staggered compression - #682
Closed
aditya1702 wants to merge 48 commits into
Conversation
The wallet_db_query_duration_seconds histogram topped out at 0.38s (ExponentialBuckets(0.0001, 2.5, 10)), so every multi-second bulk COPY landed in +Inf and histogram_quantile panels clipped at ~0.38s, under-reporting exactly the queries being optimized. Widen to ExponentialBuckets(0.0001, 3, 12) — 0.1ms through ~17.7s.
…-ledger lookup The oldest-ledger lookup (SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1) has two consumers — backfill gap detection's left bound (GetOldestLedger) and the hourly reconcile_oldest_cursor TimescaleDB job — and neither had an ordered path: the transactions migration dropped TimescaleDB's default partition-column index as consumer-less, an audit that counted WHERE-clause consumers and missed that this query consumes the index's ordering. Without it the planner pulls the first row from every chunk instead of running an ordered ChunkAppend that stops at the oldest (12-187s on an 87GB DB; the hourly job alone accounted for 45% of all DB disk-read time and flushed shared_buffers every run). Keep the default index (fresh databases get it from create_hypertable; already-migrated environments need a one-time manual `CREATE INDEX transactions_ledger_created_at_idx ON transactions (ledger_created_at DESC)`) and drop the to_id tie-break from both query sites: close times increase strictly, so rows sharing a ledger_created_at carry the same ledger_number and the tie-break could only force an incremental sort on top of the index's pathkeys. Covered by a new test that runs the reconciliation job via run_job against out-of-order chunks.
…flag ingestService carried four fields (appTracker, getLedgersLimit, knownContractIDs, contractMetadataService) and Indexer two (ingestionMetrics, networkPassphrase) that were assigned at construction and never read — consumers get these through their own constructors or through internal/ingest.Configs. Deleting them also drops AppTracker, GetLedgersLimit, and ContractMetadataService from IngestServiceConfig (the checkpoint service keeps its own metadata-service wiring). Also removes the --latest-ledger-cursor-name flag, which only fed a deprecation warning, and the phantom load_current_state phase label from the ProtocolStateProcessingDuration help text (never emitted).
Every remaining parameter of persistLedgerData had exactly one production value: cursorName was always the latest-ledger cursor (the blind-Update branch was reachable only from tests), ledgerMeta was never nil, and the numTxs/numOps returns only fed two counters the buffer already knows via GetNumberOfTransactions/GetNumberOfOperations. The signature is now (ctx, ledgerSeq, ledgerMeta, plan, contractData, buffer) error, the cursor write is unconditionally guarded, and the contractDataMemo nil-receiver branch (test-only) is gone. IndexerBufferInterface shrinks to its real seam — the indexer's fold-in plus the five getters insertIntoDB reads; everything else is reached on the concrete *IndexerBuffer. Tests now exercise the guarded cursor path persistLedgerData actually runs in production.
…Backoff persistLedgerDataWithRetry (né ingestProcessedDataWithRetry) hand-rolled the same attempt loop, exponential backoff, cap, and context handling that utils.RetryWithBackoff already provides — the body is now a single RetryWithBackoff call with isPermanentPersistError as the classifier. Metric semantics preserved: transient retries count RetriesTotal, permanent errors count ErrorsTotal and return immediately, exhaustion counts both plus RetryExhaustionsTotal, and context cancellation counts neither. RetryWithBackoff's exhaustion return now wraps the exported sentinel utils.ErrRetriesExhausted alongside the final error, which is how the wrapper tells exhaustion apart from a permanent-error exit.
…ist stages The live loop ran fetch (~0.3s of waiting), process_ledger (~0.8s of Go CPU), and the DB persist (~1.1s) strictly in sequence, so the per-ledger time was their sum — above a 2s ledger cadence at high transaction volume even though each stage individually fits. The loop is now a three-stage errgroup pipeline over depth-1 channels: while ledger N persists, N+1 processes and N+2 is fetched, making the ledger time the slowest stage instead of the sum. Persist stays strictly sequential in ledger order (the guarded cursor and per-protocol CAS chain advance N-1 → N) and is the only writing stage, so the advisory-lock liveness probe moves there. Classification planning also stays on the persist stage: its known-hash lookup is a non-transactional pool read that must observe the previous ledger's committed protocol_wasms rows. Two IndexerBuffers rotate through a freeBuffers channel, cleared only at take so a clear can never tear a persist still reading the maps the getters alias. Any stage error cancels the pipeline and the process exits, resuming from the cursor on restart — the Duration metric now sums the ledger's stage times, since wall time under a pipeline would count queue wait.
…with coordinated late commits The live persist ran its three COPY families — transactions(+accounts), operations(+accounts), state_changes — sequentially inside one transaction on one connection, so the insert phase cost their sum and a single Postgres backend did all the index maintenance. They now stream concurrently on three sibling connections, each in its own transaction, while the coordinating transaction stages everything else (assets, contracts, classification, protocol state, token changes, cursor). The table sets are disjoint with no FKs among them, and every goroutine only reads the quiescent buffer. Commits are the visibility point and are held until all four transactions have done their work: siblings commit first (sub-ms each), the coordinating transaction — whose cursor is the authority on which ledgers exist — strictly last. A failure before the first commit rolls everything back and stays retryable exactly as before; a failure after it wraps the new ErrPartialPersist sentinel, which isPermanentPersistError classifies as fatal, because COPY has no ON CONFLICT and re-running the ledger would collide on primary keys. The only crash state this ordering can produce is orphaned bulk rows for the single ledger past the committed cursor, so startup runs IngestStoreModel.DeleteRowsAboveLedger before resuming: one transaction of TOID-bounded deletes (rows of ledgers > cursor are exactly rows with TOID >= toid.New(cursor+1,0,0)) across the five bulk tables, kept chunk-local by each table's chunk skipping. Backfill keeps the single-transaction insertIntoDB path unchanged.
aditya1702
force-pushed
the
replay-loadtest-ingest-pipeline
branch
from
August 10, 2026 21:23
0d92cb5 to
0a18a10
Compare
…the schedule interval Each hypertable's columnstore policy is auto-created with an identical schedule, so all five fire at the same instant and compress their just-closed chunks concurrently — an I/O storm that starves the persist stage (measured on the loadtest rig: persist p50 0.68s -> 3.0s for 18 of every 60 minutes). Converge each policy onto a fixed schedule anchored to the interval grid with a distinct per-table slot offset, so at most one policy comes due at a time. Jobs already on their slot are left untouched across restarts, preserving next_start and run history.
…ommits When the process stage finishes ledgers faster than persist drains them, the persist stage now folds up to --live-persist-max-batch-size (default 5) consecutive ledgers into one commit set: each sibling COPY streams the whole batch on its connection, the coordinating transaction stages the batch's ledgers in order, and the commit barrier fires once — amortizing COPY setup, index-page churn, and fsyncs across the backlog. While the pipeline keeps pace every batch has size 1 and behavior is unchanged. A ledger with classification inputs always opens its own batch: its plan's pool reads see exactly what the previous batch committed, which preserves the deployed-contract-sees-prior-wasm invariant. The cursor stays the authority — one guarded update lands on the batch's last ledger, and a pre-commit failure rolls back and retries the whole batch. wallet_ingestion_persist_batch_size observes coalescing; per-ledger duration histograms record each ledger's amortized share so panels stay comparable across batch sizes.
…on WAL flush SET LOCAL synchronous_commit = off on each sibling session removes up to three serialized fsync waits per persist. Durability is unchanged: the coordinating transaction commits synchronously and strictly last, and its flush covers all earlier WAL including the sibling commit records, so a durable cursor implies durable siblings; rows a crash could lose are exactly the unacknowledged ones startup reconciliation deletes.
Each pipe's reader now buffers two decoded frames beyond the one in flight, so a writer streams its next frames through the FIFO while the consumer merges and processes earlier ledgers. With lockstep delivery every GetLedger waited on the slowest writer's in-flight frame — a constant ~0.6s/ledger at full volume that the one-frame handoff could not hide. Backpressure still bounds the writers, now with three frames of slack; an epoch's terminating error stays ordered behind its frames, so restart handling is unchanged.
…ary key TOID encoding makes a state change's parent transaction to_id derivable from its operation_id (to_id = operation_id &^ 0xFFF), so BatchGetByOperationID and BatchGetByOperationIDs now bind (ledger_created_at, to_id, operation_id) — a primary-key prefix seek — instead of filtering operation_id across every entry sharing the timestamp (measured 7 vs 307 buffers on a single-chunk fixture). That leaves idx_state_changes_operation_id with no consumer; it is removed. idx_state_changes_account_category narrows to idx_state_changes_account_id (account_id + the PK sort key): the category and reason columns only ever seeded an ordered seek for the one filter shape that pins both, while every other shape and BatchGetAccountStateChangesByToIDs seek on account_id alone. Filtered variants scan-and-filter the account's rows on the active chunk and prune compressed chunks via the existing bloom sparse indexes. Two fewer btree columns per insert on the highest-volume table. Already-migrated DBs need the one-time manual DDL (drop both old indexes, create the narrowed one); TimescaleDB rejects CREATE INDEX CONCURRENTLY on hypertables.
…lings The live persist path's sibling COPY streams grow from three to five: transactions_accounts and operations_accounts — the two largest index-maintenance payloads, each carrying an account_id-leading unique PK — stream concurrently with their parent tables instead of serially behind them. TransactionModel and OperationModel each split BatchCopy into a parent-table COPY and BatchCopyAccounts for the link table, with the link COPY's duration and batch size recorded under its own metric labels rather than folded into the parent's. Each sibling now writes exactly one table, so the disjointness at the commit barrier is per-table; the backfill path calls the five inserts in the old order and is behaviorally unchanged. Persist holds six connections at its commit barrier; the pool default rises to 12 so the persist path never queues on Acquire behind the advisory-lock session or pool-side classification reads.
… random order The native_balances and trustline_balances batch upserts arrive in Go map-iteration order and unconditionally rewrite every conflicting row. On the loadtest rig the pattern measured 60+ heap buffer touches per row (vs ~6 healthy) against a table bloated to 97% free space by the churn. Two changes: rows sort by the primary-key columns before the UNNEST arrays are built, so the btree descent and heap touches run in key order; and the DO UPDATE carries an IS DISTINCT FROM guard, so an identical row produces no new tuple version, no dead tuple, no WAL, and no index churn.
…cores runtime.NumCPU() reports the node's cores in a container with a CPU limit, so a 6-core pod on a 36-core node ran 72 indexer workers over 6 schedulable threads — pure scheduler churn on CPU-bound work. GOMAXPROCS honors the limit.
The buffer stored transactions keyed by hash while tracking their participants keyed by ToID. The streaming-loadtest backend's merged bootstrap ledgers can carry the same envelope at several tx-set positions — distinct ToIDs, one hash — and the two maps then disagreed: duplicate transactions were silently dropped from the transactions COPY while their participant links survived, whose ledger_created_at lookup missed and COPYed the zero timestamp into a year-0001 chunk (observed on the rig: 6 orphaned transactions_accounts rows, 5 missing transaction rows across bootstrap ledgers). Real networks cannot repeat a hash within or across ledgers, so both maps now share the ToID key domain by construction, mirroring the operations pair. BatchCopyAccounts on both link tables now errors on a ToID/opID with no parent row instead of silently writing a zero timestamp.
…per processor Every state-change and effects processor independently called Transaction.GetOperationChanges for the same operation, and the SDK accessor re-decodes the op's meta, re-allocates, and re-sorts the changes on every call — 19% of process-stage CPU in GetChangesFromLedgerEntryChanges, 14% in sortChanges, and a large share of GC pressure on a production profile. All of an operation's processors share one TransactionOperationWrapper, so Changes() now memoizes the extraction on the wrapper; all 14 call sites route through it, audited read-only (the memoized slice is shared — callers must not mutate it or write through Pre/Post). A wrapper copied to describe a different operation resets its memo so the cache always matches Index. BenchmarkProcessRealLedger on the five real-ledger fixtures: 40-50% less wall time, ~50% fewer bytes and ~57% fewer allocations per ledger.
aditya1702
changed the title
Batching is opt-in per deployment: at mainnet/testnet cadence the close time comfortably exceeds the persist time, so every ledger commits on its own; high-TPL/short-block deployments raise the flag to amortize backlogs.
…dger's close time DeleteRowsAboveLedger scanned every chunk TOID chunk-skipping stats could not exclude — stats only cover compressed chunks, and three of the five tables have no TOID-leading index — which took over 10 minutes on a loadtest DB carrying hours of full ledgers, long enough that the unread meta pipes wedged all three generators into liveness-probe kills. Close times are monotone, so every orphan above the cursor carries ledger_created_at at or after the cursor ledger's own close time; binding that as a partition-column predicate statically excludes all older chunks regardless of compression, and turns state_changes' scan into a primary-key range seek. A cursor ledger with no transactions row leaves the bound unresolvable and the deletes fall back to the unbounded scan.
…allel apply-load's meta write is synchronous: core does not start generating its next ledger until the consumer drains the current frame, and the reader drained only as fast as it decoded — putting XDR decode inside every producer's ledger cycle (measured: cycle = generation + decode, which capped the merged stream well under target cadence). Each pipe's reader now slurps a record's raw bytes at transfer speed and hands them to a per-pipe decode goroutine, so the producer starts its next ledger while the previous frame decodes. Frame and error ordering are preserved: a drain or decode error is always the last element delivered and ends the epoch exactly as before.
The metadata fetcher's fetched/failure caches are in-memory only, so a token whose contract_tokens row already carries metadata was still re-fetched over RPC — once per process lifetime after every restart on a real network, and forever on a 5-minute backoff against a deployment whose RPC can never resolve metadata (the loadtest rig's dead endpoint with externally seeded rows), where each retry burned the ~850ms simulate backoff ladder inside prepare_classification. After Apply persists a batch's rows, contracts whose row has metadata are marked fetched, making the database the durable cache. Tokens whose metadata is genuinely missing stay unmarked and keep the existing backoff retry.
The classification-safety cut isolated any ledger carrying protocol wasm/contract observations, but a plan's pool reads are only unsound for inputs no committed batch has classified yet — and re-observations of known contracts are the overwhelmingly common case (synthetic loadtest traffic re-observes the same token contracts every ledger, which cut every batch to size one and disabled batching entirely; goroutine dumps showed the process stage starved of buffers held by the always-cut pending queue). The persist goroutine now keeps seen-sets of wasm hashes and contract IDs folded in after each successful commit; only a ledger introducing unseen inputs opens its own batch. A rolled-back batch marks nothing, and a restart starts empty — conservative until re-warmed.
The seen-set comment block split the struct's alignment group, leaving the fields below it on the wider alignment. golangci-lint's gofmt check fails on it; `make check` cannot catch this because its `tidy` step rewrites the file before the `fmt` gate reads it.
…eachable A buffer stays checked out from the moment the process stage takes it until the batch carrying it commits, so the rotation needs 2*cap buffers to sustain a full batch: cap for the batch in flight, cap-1 queued in processed, one being filled. It held cap+1, so process starved on freeBuffers after filling (cap+1)-k and the batch settled at the fixed point k = (cap+1)-k. The batch size was therefore pinned below the configured cap at every load — loadtest measured exactly 3.00 against a cap of 5 across 17 sample windows spanning 3.9k to 9.8k tx/s, and the new test measures 2.04 against a cap of 3 when the pool is sized the old way.
…e bar The phase histogram stepped 0.5 -> 1 -> 2 seconds, so a stage sitting at 0.9s and a stage sitting at 1.9s both reported the same p99 bucket. The pipeline's contract is that the slowest stage's p99 stays under the ledger close time, so the metric could not grade the thing it exists to grade. Boundaries now land exactly on 0.6s (Phase-3 block rate) and 1s (the loadtest rig's 1.67x-sized merged ledgers), with 0.5 and 0.75 around them.
Pairing metas with envelopes needs every envelope's hash, and building it was the pipeline's largest serial stretch: on the loadtest rig it ran ~0.37s single-threaded ahead of the fan-out, 41% of the process stage, of which 85% was the hashing itself. Reading transactions out of a LedgerCloseMeta now hashes envelopes across the indexer's worker pool instead of on the caller's goroutine. Three costs went with it. The network id is a hash of the passphrase alone, so it is computed once per ledger rather than once per transaction. Each chunk marshals through one xdr.EncodingBuffer, which reuses its scratch space across transactions instead of allocating a buffer per hash — this path alone was 8.5% of everything the ingest process allocated. And the envelope's transaction is tagged in place rather than copied by value. On the real pubnet fixtures, where the fan-out barely engages at a few hundred transactions per ledger: 2.8-3.8x faster, 4.3-8.6x fewer allocations. Also restores the independence of the two reader-based oracles in the test suite. They materialized their transactions through the same function they were the oracle for, so they now go through getLedgerTransactionsViaReader, which drives the SDK reader and shares no code with the fan-out. That helper is the merge gate for this change too, via TestGetLedgerTransactions_EquivalenceOnRealLedgers.
isLiquidityPool and isClaimableBalance ran a full strkey decode — base32, CRC16 validation, and a byte copy of the input — on both endpoints of every transfer event. Those endpoints are ordinary account and contract addresses almost every time, so nearly all of that work only ever concluded "no". A strkey version byte is a multiple of 8, so its top five bits are exactly what the first base32 character encodes: only an 'L' can decode to a liquidity pool and only a 'B' to a claimable balance, whatever the payload. Checking that character first skips the decode for everything else, and a string that passes it still goes through the decode, so validation is unchanged. Together this was 13.6% of the token-transfer processor on the loadtest rig.
aditya1702
force-pushed
the
replay-loadtest-ingest-pipeline
branch
from
August 11, 2026 22:14
344e81b to
0c05682
Compare
…wrapper memos The persist stage extracted each ledger's ContractData changes through tx.GetChanges(), which rebuilds and ledger-key-sorts every operation's change group — work the process stage had already done and memoized on the shared TransactionOperationWrapper. On the loadtest rig that duplicate path was 8.7s of a 239.7s profile (3.6% of ingest CPU, plus its allocation/GC halo), all of it on the serial persist goroutine. processTransaction now collects each successful transaction's ContractData changes from the wrappers' memoized Changes() slices, mirroring GetChanges' composition per meta version (tx-level before, operations ascending, tx-level after); transaction-level segments are only materialized when a type-tag scan of the raw group mentions ContractData at all. Results fold into the ledger buffer in transaction order and reach the persist stage on the buffer it already carries, replacing the contractDataMemo and the transactions plumbing through processedLedger; ProcessLedger no longer returns the materialized transactions. Merge gate: TestProcessLedgerTransactions_ContractDataChangesMatchReaderExtraction compares the buffer's collection against the reader-based tx.GetChanges() reference over every real-ledger fixture; synthetic tests pin the success gate, the meta-version guard, wrapper-memo serving, and the no-wrapper fallback.
AddressByteaMemo caches the 33-byte form per unique address string within one COPY build, where addresses repeat heavily (a token's transfers share token_id, an account's rows share account_id). A hit skips the strkey decode and the 33-byte allocation entirely: 278ns/200B/4allocs → 9.3ns/0B/0allocs. AddressBytea.Value now rejects payloads that are not exactly 32 bytes instead of silently truncating them (an M-address decodes to 40 bytes; unreachable from current callers, defensive).
dedupeParticipants keyed a map on id.Address(), strkey-encoding every participant just to throw the string away — the caller immediately re-encodes the survivors. It now compares the raw ed25519 keys with a linear scan over the kept prefix, filtering in place: zero allocations, zero encodes, and the output keeps deterministic first-seen order instead of map order (downstream folds into a set, so ordering was and stays unobservable). The replaced implementation stays in the test file as a differential oracle.
GetOperationsParticipants hoists the envelope operations and ledger close time out of the per-op loop, sizes its result map, and drops the unreachable merge branch (opID embeds the operation index, so each iteration writes a distinct key). Per-op participant sets are pre-sized. The metric-only unique-participants map is gone: the tx-level participant set doubles as the cardinality accumulator once its slice snapshot is taken.
…ator The Soroban participant walk allocated a set per processor, per auth entry, and per invocation-tree node, merging each level with Union (which allocates a third set per call), and re-encoded the op source address in every sub-processor. The whole path now adds into the single set GetOperationParticipants already built: no intermediate sets, no Unions, one source encode per op. Riding along, profile-guided: - The per-node reflective utils.IsEmpty guard (which heap-escapes the whole invocation) is a typed nil-arm check: a zero-value invocation reports the ContractFn arm with a nil pointer, which GetContractFn would dereference. - Sub-processors fetch MustInvokeHostFunctionOp once for both the host-function args and the auth entries (was two to three union-arm copies per op). - Invocation nodes and auth entries are visited by pointer instead of by value. Behavior is pinned by the existing exact-set expectations plus new tests for the zero-value invocation and the muxed-source M-address form.
Every state_changes row decoded up to six strkey addresses from scratch, and the values repeat heavily within one batch: every transfer of a token shares its token_id and an account's fee/balance/effect rows share its account_id. One AddressByteaMemo per BatchCopy call now serves all six columns, replacing the pgtypeBytesFromNullAddressBytea helper whose only caller this was.
Participants are deduplicated per transaction/per operation upstream, so a busy account is decoded once per tx or op it touches in transactions_accounts and operations_accounts. One AddressByteaMemo per BatchCopyAccounts call collapses that to one decode per unique address per batch.
SourceAccount() returned a pointer, heap-escaping a stack local on every call in the common no-explicit-source branch — and it is called several times per operation across the participants and effects paths. It now returns the xdr.MuxedAccount by value; callers that need a pointer or addressability take a local first.
TransactionOperationWrapper copied the whole ingest.LedgerTransaction (envelope, result, meta, fee changes, ledger header) into every per-operation wrapper. The field is now a pointer: GetOperationsParticipants materializes the transaction once and every wrapper of the ledger transaction aliases it. The wrapper path is read-only, and each per-transaction worker owns its transaction, so nothing is shared across goroutines. makeFeeBumpOp (test util) clones the transaction before rewriting the envelope so a fee-bump variant cannot write through the base op's now-shared pointer.
The streaming-loadtest backend's meta sources generalize beyond FIFOs: a tcp-listen://HOST:PORT entry binds a listener eagerly at construction and serves each producer connection as one stream epoch, so apply-load producers can run in their own pods and dial in. Connection close is the epoch boundary (as FIFO EOF is), the listener outlives epochs to serve producer restarts, and keepalive surfaces a vanished peer as a read error. Frame draining, decoding, renumbering, and merging are shared between both source kinds; entry order still defines merge order. The flag is now --loadtest-meta-sources / LOADTEST_META_SOURCES.
…ling The native-balance and liquidity-pool upserts ran serially inside the coordinating transaction, lengthening its critical path while the COPY siblings streamed concurrently. Their tables carry no foreign keys, so a sixth sibling now carries them; the trustline and SAC balances stay on the coordinating transaction because their parent rows (trustline_assets, contract_tokens) are staged uncommitted there, and a sibling committing first would fail FK checks against committed state. Reconciliation (DeleteRowsAboveLedger) does not cover the balance tables: a crash can leave balance rows above the committed cursor, which is safe because the upserts are idempotent and reapply when those ledgers re-ingest.
AssignStateChangeOrdinals runs four times per transaction, and its group counter map — sized to the stream — was allocated and discarded on every call. Streams are a handful of changes, so counting earlier same-group elements with a quadratic scan is cheaper and allocation-free, and produces byte-identical IDs.
…nge path - Sponsorship scans allocate their detail maps only after detecting a transition; accounts with no sponsored signers skip the SDK's SponsorPerSigner map-and-slice build entirely. - EffectOutput drops five write-only fields (EffectID, EffectIndex, LedgerClosed, LedgerSequence, AddressMuxed) and the per-effect fmt.Sprintf that populated them; nothing reads them. - Transaction hashes are hex-encoded at the rare log/error sites instead of eagerly per operation. - Soroban-only processors check the operation type before installing their duration-metric closure, so classic operations pay neither the timestamps nor the label lookup; invalid-op-type calls no longer land in the histogram. - ContractDeploy's dedupe map allocates on first deploy, not per call. - Zero-effect operations return before the master builder is built.
Deriving a SAC contract ID costs a SHA-256 over the asset's contract-ID preimage plus a strkey encode, and both the effects and token-transfer processors recompute it for the same few assets on every event. Each processor now memoizes by asset identity; entries are content-derived under the processor's fixed network passphrase, so they never invalidate, and sync.Map keeps the lookup safe across the indexer's pool workers.
Each buffered lookahead frame is a fully decoded LedgerCloseMeta — tens of MB of pointer-dense XDR per source, ~600MB resident across 12 sources per lookahead unit — and that mass is scanned by every GC cycle. One frame of lookahead still overlaps the writer's streaming with the consumer's processing; the second bought no cadence and cost scan time.
The sponsorship effect paths had no coverage: no test exercised addSignerSponsorshipEffects, addLedgerEntrySponsorshipEffects, or the accountHasSponsoredSigner guard that spares the SponsorPerSigner build for accounts sponsoring nothing. Covers every transition (created, removed, updated, unchanged, whole-entry removal), the guard's over-admit-never-over-reject contract, deterministic signer ordering, and each sponsorable entry type's details shape.
Every NewStateChangeBuilder and Clone() heap-allocated a ~480-byte builder, and the hot paths clone constantly — a plain payment paid four heap builders for three state changes. The builder is now used by value: With* methods mutate their own copy and return it, branching is plain assignment, Clone() is gone, and builders stay on the stack. The write-only metricsService field is dropped from the builder along with the constructor parameter. parseTrustline now returns the builder it produced so the balance-authorization change emitted for a new trustline keeps inheriting the resolved token and the trustline limit exactly as it did through the shared pointer; whether that inherited limit belongs on a BALANCE_AUTHORIZATION row is flagged in the PR as a follow-up decision. Helper signatures across the effects, token-transfer, contract-deploy, SAC, SEP-41, and Blend processors take the builder by value; output is byte-identical everywhere.
The AddressMuxed removal left github.com/guregu/null unused module-wide, and the sponsorship fixtures always build entries for sponsoredAccount, so the account parameter comes off both helpers (unparam).
A pool-share balance is read alongside its pool's reserves, and ingestion writes both tables in one transaction with the pool row first, so a balance whose pool is missing can only be an ingestion bug. The new foreign key turns that bug into a loud write-time failure, matching the trustline_balances -> trustline_assets constraint; deferred so the commit-time check covers same-transaction pool deletions. Verified orphan-free on the dev and loadtest databases before adding.
…bling Each balance family now rides the transaction that stages its foreign-key parents: trustline_assets and trustline_balances move from the coordinating transaction onto a new sibling, alongside the balances sibling that carries pools and pool-share balances under the new pool foreign key. Every FK is checked at its own transaction's commit, and the coordinator's serial path shrinks to contracts, classification, protocol state, SAC balances, and the cursor. SAC balances stay coordinated: their parent (contract_tokens) is also written by the classification path there, and a same-key insert from two concurrent transactions could deadlock at the commit barrier.
…tion XDR encoder The signers-only skip asked the SDK's AccountChangedExceptSigners, which deep-copies both account entries, XDR-marshals each, and compares the bytes — 2.6% of ingest CPU and 5.7% of allocated bytes at ~18k tx/s. A local field comparison preserves the exact semantics (only the Signers slice is ignored; NumSubEntries and V2 signer-sponsoring IDs still count; a missing extension equals V1 with zero liabilities), pinned by a 25-case differential test against the SDK function as oracle. The always-nil error plumbing collapses with it. ConvertOperation's per-operation XDR marshal allocated a fresh encoder and growth-doubled buffer each call; a pooled EncodingBuffer reuses the scratch across pool workers and allocates only the retained, exact-size copy of each operation's bytes.
This was referenced Aug 13, 2026
Open
Contributor
Author
|
Replaced by a reviewable split — every hunk of this PR is preserved exactly (the recombined branches reproduce this PR's tip
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rearchitects live ingestion for sustained high-TPL ledgers: a fetch ‖ process ‖ persist pipeline with bounded batched commits, seven parallel sibling write streams (five COPY families plus two balance siblings, each carrying its foreign-key parents), once-per-operation change decoding, an allocation-lean state-change build (by-value builder, sponsorship early-outs, memoized asset→contract-ID derivation, map-free ordinal assignment), allocation-lean participant extraction, memoized address→BYTEA conversion in the COPY builds, staggered TimescaleDB compression, and a consumer-justified index diet on
state_changes. Also fixes a silent transaction-loss bug in the indexer buffer, hardens the balance upserts against heap bloat, adds the missing sponsorship-effect test coverage, and generalizes the dev-only loadtest backend's meta sources to TCP listeners so the apply-load producers can run in their own pods, off the measured pod's node.Why
The network roadmap shrinks block close time 2s → 1s → 600ms while per-ledger transaction volume stays roughly constant, so live ingestion must process one full ledger in under the block time. The engineering contract is p99 of the slowest pipeline stage ≤ block time. On the loadtest rig (Phase-1 shape: 6000 SAC + 1000 classic + 4000 custom-token + 1500 soroswap TPL), serial ingestion took 7.7–8.4 s/ledger. The changes here address, in measured order of impact: serialized insert streams, the hourly compression I/O storm (persist p50 0.68 s → 3.0 s for 18 of every 60 minutes), redundant per-processor change decoding (19% of process CPU in
GetChangesFromLedgerEntryChanges, 14% in the SDK'ssortChanges), and index maintenance on the highest-volume table.How
Pipeline (fetch ‖ process ‖ persist)
ingestLiveLedgersruns three stages connected by channels: while ledger N persists, N+1 processes and N+2 is fetched. Ledger time becomes the slowest stage, not the sum. Persist stays strictly sequential in ledger order (guarded cursor and per-protocol CAS chains advance N-1 → N); any stage error cancels the pipeline and the process resumes from the cursor after restart.wallet_ingestion_duration_secondssums the stage times so panels stay comparable to the old serial values; throughput lives inrate(wallet_ingestion_ledgers_total)and per-stage numbers inwallet_ingestion_phase_duration_seconds{phase}, whose buckets bracket the contract: boundaries sit on 0.6s (the 600ms block target) and 1s (the rig's 1.67×-sized merged ledgers), with 0.5 and 0.75 around them. The previous 0.5 → 1 → 2 steps could not separate a stage at 1.05s from one at 1.95s, so the metric could not grade the bar it exists to measure.Five sibling COPY streams + coordinated late commits
Each persist opens five sibling transactions —
transactions,transactions_accounts,operations,operations_accounts,state_changes, each owning exactly one table — plus a coordinating transaction for everything else (assets, contracts, classification, protocol state, token changes, cursor). All the slow work happens uncommitted; commits fire only after every stream succeeds, siblings first, coordinator strictly last. The cursor is the authority: the only crash state is orphaned bulk rows above the committed cursor, whichDeleteRowsAboveLedgerremoves at startup. Failure before the first commit is retryable; after it,ErrPartialPersistis fatal.Sibling sessions run
SET LOCAL synchronous_commit = off: the coordinator's synchronous, strictly-last commit flushes all earlier WAL including the sibling commit records, so a durable cursor implies durable siblings — up to five fsync waits per persist removed with no durability change.Bounded batched commits (
--live-persist-max-batch-size, default 1)When process finishes ledgers faster than persist drains them, persist coalesces up to N consecutive ledgers into one commit set — each sibling streams the whole batch, the coordinator stages ledgers in order, the cursor lands on the batch's last ledger — amortizing COPY setup, index churn, and fsyncs across the backlog. While the pipeline keeps pace every batch has size 1 and behavior is exactly the unbatched persist. Batching is opt-in per deployment: the default 1 keeps commit-per-ledger on mainnet/testnet, whose 5s close time comfortably exceeds the persist time; high-TPL/short-block deployments (the loadtest rig, future network phases) raise it. A ledger carrying classification inputs no committed batch has yet classified opens its own batch, preserving the plan's requirement that its pool reads see the previous commit; re-observations of already-classified contracts ride mid-batch.
wallet_ingestion_persist_batch_sizeobserves coalescing; per-ledger histograms record amortized shares.The cap also sizes the process↔persist buffer rotation. A buffer stays checked out from the moment process takes it until the batch carrying it commits, so sustaining a full batch needs 2N+1 buffers: N held by the batch in flight, N-1 queued behind it, one being filled. Sized any tighter, process starves on free buffers before the queue can refill and the batch settles at a fixed point strictly below N — the configured cap becomes unreachable at every load, which the histogram reports as a suspiciously constant value. A test drives the pipeline with the process stage ahead and fails if the mean batch falls short of the cap. Each buffer retains a merged ledger's maps, so raising N raises ingest's memory floor with it.
Process stage: decode once, hash in parallel
Every state-change/effects processor called
Transaction.GetOperationChangesindependently, and the SDK re-decodes, re-allocates, and re-sorts on each call. All of an operation's processors share oneTransactionOperationWrapper, soChanges()now memoizes the extraction; all 14 call sites route through it (audited read-only — the memoized slice is shared).BenchmarkProcessRealLedgeron five real pubnet ledgers: 40–50% less wall time, ~50% fewer bytes, ~57% fewer allocations. The ledger-indexer pool also sizes offGOMAXPROCSinstead of node cores (a 6-core pod on a 36-core node ran 72 workers over 6 schedulable threads).Reading a ledger's transactions was what remained of the stage's serial time. Pairing metas with envelopes needs every envelope's hash — the transaction set carries envelopes in agreement order while metas are sorted by hash — and the SDK reader computes all of them on the caller's goroutine before returning the first transaction, so no amount of overlapping with the fan-out that follows can recover it. On the rig that was ~0.37 s single-threaded per ledger, 41% of the process stage. Hashing now runs across the indexer's worker pool, one contiguous chunk per schedulable thread, and three costs went with it: the network id is a hash of the passphrase alone, so it is computed once per ledger instead of once per transaction; each chunk marshals through one reused
xdr.EncodingBufferrather than allocating a buffer per hash — that path alone was 8.5% of everything the ingest process allocated, against a GC taking 25.7% of its CPU; and the envelope's transaction is tagged in place instead of copied by value. On the real pubnet fixtures, where a few hundred transactions per ledger barely engage the fan-out,BenchmarkGetLedgerTransactionsis 2.8–3.8× faster with 4.3–8.6× fewer allocations. A differential test against the SDK reader over every committed fixture is the merge gate, and it also restores the independence of the two other reader-based oracles in the package, which had been materializing their transactions through the very function they were the oracle for.The wrapper memo also feeds ContractData extraction now. The persist stage had been deriving each ledger's ContractData changes through
tx.GetChanges(), which rebuilds and ledger-key-sorts every operation's change group from the meta — the same work the wrappers had already done and memoized one stage earlier. Measured on the rig, that duplicate path was 3.6% of ingest CPU plus its allocation halo, all of it on the serial persist goroutine.processTransactionnow collects each successful transaction's ContractData changes from the memoized slices, mirroringGetChanges' per-meta-version composition (transaction-level before, operations ascending, transaction-level after), and folds them into the ledger buffer persist already receives — replacing the lazy extraction memo and the transactions plumbing through the process→persist handoff. Transaction-level segments are materialized only when a type-tag scan of the raw group mentions ContractData at all. The reader-basedtx.GetChanges()reference stays the merge gate over every committed fixture.Graded on the rig with reset-matched fresh-DB windows:
GetChangesvanishes from the live profile,operationChangesfalls 5.46% → 2.68% of ingest CPU, and the collection itself costs 0.96%. The wall-time effect is a stage rebalance — persist mean 0.956s → 0.791s while process absorbs ~the same amount — which is the right trade: the contract grades the slowest stage, and the stages now sit nearly level instead of persist-dominated.isLiquidityPoolandisClaimableBalancestop decoding too. They ran a full strkey decode — base32, CRC16, and a copy of the input — on both endpoints of every transfer event, which are ordinary account and contract addresses almost every time. A strkey version byte is a multiple of 8, so its top five bits are exactly what the first base32 character encodes, and only anLor aBcan decode to those two version bytes whatever the payload. Testing that character first skips the decode for everything else while anything that passes it still goes through the full decode, so validation is unchanged. 13.6% of the token-transfer processor on the rig.Participant extraction without the churn
With persist capped by the 15-minute chunks (below), process became the binding stage, and its largest attackable block was
GetOperationsParticipants— 6.8% of ingest CPU and 8.2% of every byte the process allocates, most of it structural rather than essential:dedupeParticipantskeyed a map onid.Address()(a full strkey encode per participant) and the caller immediately re-encoded every survivor. It now compares raw ed25519 keys with a linear scan over the kept prefix, filtering in place — zero allocations, zero encodes, deterministic first-seen order (downstream folds into a set, so ordering was and stays unobservable). The replaced implementation lives on in the test file as a differential oracle.Union— which allocates a third set per call — plus a fresh encode of the op source in every sub-processor. The whole path now adds into the single setGetOperationParticipantsalready built: no intermediate sets, no merges, one source encode per op. Sub-processors fetchMustInvokeHostFunctionOponce for both the args and the auth entries, and invocation nodes are visited by pointer.utils.IsEmptyheap-escapes the wholeSorobanAuthorizedInvocationand deep-walks it via reflection, per node. What it actually protects against is precise: a zero-value invocation reports the ContractFn union arm with a nil pointer, whichGetContractFnwould dereference. The replacement is a typed nil-arm check — same protection, no reflection — pinned by a new zero-value-invocation test.SourceAccount()returned a pointer, heap-escaping a stack local on every call in the common branch; it returns by value now. Each per-op wrapper copied the wholeingest.LedgerTransaction; wrappers of a transaction now share one pointer (the path is read-only, and each per-tx worker owns its transaction —-raceclean). Loop hoists, map size hints, and the metric-only unique-participants map folded into an existing set round it out.BenchmarkProcessRealLedgerover the committed pubnet fixtures: −5.4% to −12.5% wall, −3.2% to −3.9% allocations. Those fixtures are classic-op-heavy, so they exercise the dedupe/struct wins but barely touch the Soroban path where the set churn concentrates; the rig's SAC-dominated ledgers are the workload this targets, graded there by profile share.Memoized address→BYTEA conversion in the COPY builds
AddressBytea.Value— a full strkey decode (base32 + CRC16) plus a 33-byte allocation per call — was 4.2% of ingest CPU and 8.6% of all bytes allocated, all on the persist side: 5.3 points of those allocations are the SDK minting a fresh base32 encoder object per decode insidestrkey.decodeString. The values repeat heavily within one COPY: every transfer of a token shares itstoken_id, an account's fee/balance/effect rows share itsaccount_id, and participants (deduplicated per-tx/per-op upstream) recur once per tx/op they touch.Each batch-build call (
state_changesBatchCopy, bothBatchCopyAccounts) now creates oneAddressByteaMemo— a plain per-call map, single-goroutine by construction, no locks, no lifetime beyond the COPY — and converts each unique address once. A hit returns the shared 33-byte slice: 269 ns / 200 B / 4 allocs → 9.2 ns / 0 B / 0 allocs.BenchmarkStateChangeModel_BatchCopyallocates 41–48% fewer bytes across batch sizes (its fixture repeats addresses the way real ledgers do); the operations-accounts bench, whose synthetic fixture mints a unique random address per row, pays the memo's miss-only overhead instead — the honest worst case, not the expected one.Riding along:
AddressBytea.Valuenow rejects payloads that are not exactly 32 bytes instead of silently truncating them (an M-address decodes to 40 bytes; unreachable from current callers, defensive).Upstream follow-ups recorded, not taken here:
strkey.decodeStringshould use its own package-level encoder instead of allocating one per decode, andAccountId.GetAddressdoes a pointless 32-byte scratch copy per call — both SDK-side, both sidestepped in-process by the memo and the encode-once dedupe.Balance siblings: each family rides with its foreign-key parents
The balance upserts ran serially inside the coordinating transaction. They now stream on two additional siblings, organized so every foreign key is checked within one transaction's own commit: a balances sibling carries native balances plus liquidity pools and pool-share balances (pools upserted first — a new FK constrains
liquidity_pool_balances.pool_id → liquidity_pools, matching the trustline constraint; verified orphan-free on the dev and loadtest databases before adding), and a trustlines sibling carries thetrustline_assetsinserts together with thetrustline_balancesupserts they parent. The SAC balances stay on the coordinator deliberately: their parent (contract_tokens) is also written by the classification path there, and splitting those writers across concurrent transactions could deadlock at the commit barrier on a same-key insert — the coordinator's serial path shrinks to contracts, classification, protocol state, SAC balances, and the cursor.Two properties worth stating explicitly: startup reconciliation (
DeleteRowsAboveLedger) does not cover the balance tables, which is safe because the upserts are idempotent and reapply when the orphaned ledgers re-ingest; and the honest sizing — measured per-ledger DB time puts the coordinator's old serial path at 0.520s against the widest COPY sibling's 0.700s, so this shortens the coordinator without moving today's persist wall-clock. It removes the coordinator from contention as balance cardinality grows (the native upsert scales with account count) and gives every balance family the same FK-with-its-parents shape.State-change build without the churn
At ~19k tx/s the
getTransactionStateChangessubtree was 23.1% of every byte ingest allocated, and most of it was structural:New/Clone— a plain payment paid four heap builders to emit three state changes.StateChangeBuilderis now used by value:With*methods mutate their own copy and return it, branching is plain assignment,Clone()is gone, and builders stay on the stack. The write-onlymetricsServicefield came off the struct. One aliasing dependency surfaced and is preserved byte-identically: the BALANCE_AUTHORIZATION change emitted for a new trustline inheritstrustline_limit_newfrom the trustline change it accompanies (parseTrustlinereturns its builder to keep that). Flagged as a follow-up decision — that field arguably has no business on an authorization row, but dropping it is a behavior change and this PR grades against byte-identical output.SponsorPerSignerbuilt a map and slice per account change even for the overwhelming majority of accounts sponsoring nothing. An in-place predicate overSignerSponsoringIDs(over-admits, never over-rejects) early-outs before anything allocates. This path had zero test coverage; it now has 28 cases covering every transition, attribution, and detail shape.EffectOutputcarried five write-only fields (EffectID— a per-effectfmt.Sprintf—EffectIndex,LedgerClosed,LedgerSequence,AddressMuxed), verified unread and removed, which also freed the module ofguregu/null.sync.Map, since one processor serves every pool worker).AccountChangedExceptSignersdeep-copies both account entries, XDR-marshals each, and compares bytes — 2.6% of ingest CPU and 5.7% of allocated bytes. A local field comparison preserves the exact semantics (only the Signers slice is ignored;NumSubEntriesand the V2 signer-sponsoring IDs still count; a missing extension equals V1 with zero liabilities), pinned by a 25-case differential test against the SDK function as oracle.ConvertOperationallocated an encoder and growth-doubled buffer per operation for the retained XDR bytes; a pooledEncodingBufferreuses the scratch across pool workers and allocates only the exact-size copy that the row keeps (1.8% CPU, 3.9% of bytes).ContractDeploy's dedupe map allocates on first deploy; zero-effect operations return before the master builder exists.BenchmarkProcessRealLedgerover the committed pubnet fixtures: −4.2% to −6.5% wall, −5.0% to −5.6% allocations, −3% bytes (all p=0.002, n=6). Real ledgers are diverse; the rig's SAC-transfer-dominated load concentrates on exactly the token-transfer path where 77% of this churn lived, so the rig's share is expected to be larger, graded there by profile delta.Compression staggering
Each hypertable's columnstore policy is auto-created with an identical schedule, so all five fired at the same instant and compressed ~an hour of chunks concurrently — measured 4.5× persist degradation for 18.5 min of every hour on the rig.
configureHypertableSettingsnow converges each policy onto a fixed schedule anchored to the interval grid with a distinct per-table slot (initial_start = date_bin(interval, now(), epoch) + interval + i/5·interval), so at most one policy comes due at a time. Jobs already on their slot are left untouched across restarts.Index diet on
state_changesTOID encoding derives a state change's parent transaction (
to_id = operation_id &^ 0xFFF), soBatchGetByOperationID(s)now bind(ledger_created_at, to_id, operation_id)— a PK-prefix seek, measured 7 vs 307 buffers — andidx_state_changes_operation_idis dropped.idx_state_changes_account_categorynarrows toidx_state_changes_account_id(account_id + the PK sort key): category/reason filters scan-and-filter the account's rows on the active chunk and prune compressed chunks via the existing bloom sparse indexes. Two fewer btree columns per insert on the highest-volume table. Already-migrated DBs need a one-time manual DDL (drop both old indexes, create the narrowed one; TimescaleDB rejectsCREATE INDEX CONCURRENTLYon hypertables).Correctness fixes
ledger_created_at(materializing a year-0001 chunk). Both maps now share the ToID key domain, mirroring the operations pair;BatchCopyAccountserrors on a link row with no parent instead of writing a zero timestamp. Real networks cannot repeat a hash, so this was latent there and triggered only on the rig.reconcile_oldest_cursorjob had been 45% of all loadtest-DB disk reads; the defaulttransactions_ledger_created_at_idxis no longer dropped and the deadto_idtie-break is removed.native_balances,trustline_balances): rows sort by PK before UNNEST (map-iteration order scattered thousands of random btree/heap probes; measured 60+ heap buffer touches/row vs ~6 healthy) and the DO UPDATE carries anIS DISTINCT FROMguard so identical rows produce no dead tuple, WAL, or index churn.DeleteRowsAboveLedgerscanned every chunk TOID chunk-skipping stats could not exclude (stats only cover compressed chunks; three of five tables have no TOID-leading index) — over 10 minutes on a DB carrying hours of full ledgers, long enough that unread meta pipes wedged the loadtest generators into liveness-probe kills. Close-time monotonicity makes every orphan'sledger_created_at≥ the cursor ledger's own close time, so the partition-column bound statically excludes all older chunks and turnsstate_changesinto a PK range seek.contract_tokensrow already carries metadata is never re-fetched over RPC (previously once per process lifetime per token after every restart, and forever on a 5-minute backoff against an RPC that can never resolve it); genuinely missing metadata keeps the existing backoff retry.utils.RetryWithBackoff.Loadtest backend
Per-source readers split into a raw drain and a parallel decoder: the drain slurps each framed XDR record at transfer speed (apply-load's meta write is synchronous, so core cannot start generating its next ledger until the frame drains — decoding inline put the XDR decoder inside every producer's ledger cycle), and a per-source goroutine decodes from memory, buffering two frames ahead. Frame and error ordering are preserved; producer restarts roll epochs exactly as before.
The backend's meta sources also generalize beyond FIFOs: a
tcp-listen://HOST:PORTentry binds a listener eagerly at construction and serves each accepted producer connection as one stream epoch, so the apply-load producers can run in their own pods and dial in. This exists because colocating 12 burstable producers with the measured ingest container is a rig artifact — under node saturation CFS weights by requests, giving the producers ~60% of the node — and prod ingest pods have no producers. Connection close is the epoch boundary exactly as FIFO EOF is, the listener outlives epochs to serve producer restarts, keepalive surfaces a vanished peer as a read error, and the frame drain/decode/renumber/merge path is shared between both source kinds; entry order still defines merge order. Backpressure semantics carry over: a frame (tens of MB) dwarfs the TCP socket buffers, so core's synchronous meta write still blocks on the consumer's drain rate, within one in-flight frame like the FIFO's kernel buffer. The flag is now--loadtest-meta-sources(envLOADTEST_META_SOURCES), replacing--loadtest-meta-pipe-paths; FIFO paths remain valid entries (tests and local runs use them). The producer side — a dialer shim that parks the connected socket on fd 3 and execs core withMETADATA_OUTPUT_STREAM="fd:3", per-ordinal startup jitter to desynchronize the OOM epoch rolls, StatefulSets, anti-affinity — lives entirely in the kube repo.The per-source lookahead also halves (
sourceLookaheadFrames2 → 1). Each buffered frame is a fully decodedLedgerCloseMeta— tens of MB of pointer-dense XDR per source, ~600MB resident across 12 sources per lookahead unit — and the GC'sscanobjectwalks that mass on every cycle (GC was 27.6% of ingest CPU at ~19k tx/s,scanobject26.8%). One frame of lookahead still overlaps the writer's streaming with the consumer's processing; the second bought no cadence and cost scan time.Measured results (loadtest rig, Phase-1 shape)
state_changesCOPY build, bytes allocatedmax_wal_size32GB (kube-side)The two rig windows are phase-matched — same 10 minutes past the hour, so the same age of the
one-hour active chunk and the same overlapping compression policy — because that turned out to
matter more than anything measured here. Persist swings 0.789 s to 3.369 s within a single
hour purely as the active chunk's indexes grow and then reset on the roll, with batching rising
to the cap and the rig flipping from supply-bound to sink-bound at the top of the ramp. Any
measurement on this rig that does not control for chunk phase is dominated by a 4.3× effect.
That argument was then tested directly: the rig's chunk interval and compression schedule moved
to 15 minutes (a deployment config change, not code in this PR), graded with fresh-DB
reset-matched windows on both sides. Persist p99 fell 1.92 s → 1.22–1.40 s and stays in that
band across chunk ages instead of tripling — the sawtooth is capped at a quarter of its former
amplitude, with p50/p90 under 1 s. Compression ratios were invariant (state_changes 6.07 → 6.13,
every table marginally better), sub-hour range queries prune to exactly the chunks they need,
and the one open read-side question is the sparse-account chunk walk at matched data age
(structurally 4× the chunks; a
LIMIT-unfilled account walks all of them). With persist capped,the process stage (p99 ≈ 1.48 s) became the binding stage against the 1 s rig-equivalent
bar; its largest attackable block on that profile —
GetOperationsParticipantsat 6.8% ofingest CPU plus the
AddressBytea.Valuedecode at 4.2% CPU / 8.6% of allocations on the persistside — is what the participants and address-memo rounds above attack. GC was 20–28% of ingest
CPU across these profiles, so the allocation cuts have paid roughly double their direct share
twice already on this rig.
The producer split then graded at 18,974–20,267 tx/s (vs 11,548–12,268 pre-split) at the
producers' 1s herder floor, with process p99 unchanged at +54% load — confirming producer
contention was never the process tail; GC pointer mass is (gcDrain share invariant at 27.6%
pre/post). At the new +50% write volume persist regressed to p99 1.82–1.85s, and a
high-resolution diagnostic attributed the pressure: WAL runs at 362 GB/h (100.6 MB/s
sustained, 80% of the WAL volume's throughput ceiling) and every checkpoint is size-triggered,
back-to-back — one per ~87s, each starting seconds after the previous completes — a
full-page-write feedback loop where continuous checkpointing keeps every page recently
checkpointed and FPWs never stop.
max_wal_size16→32GB (a deployment config change in thekube repo, not code in this PR) breaks the loop from both ends. The checkpoint-interval doubling alone took persist p99 1.85 s → 1.48 s in a graded
16-minute window. A subsequent 5-second-resolution co-occurrence analysis attributed the
residual tails end-to-end: the persist 1.0–1.5 s tail concentrates within ±10 s of checkpoint
completions (4.2× lift, decaying to 1.1× at ±60 s; compression jobs show no enrichment), and
the process >1 s tail follows persist-tail intervals at 10–11× lift — buffer-rotation
backpressure, not process work. (An earlier "correlated ~1.9 s COPY stall" reading of the
per-query histograms was a bucket-interpolation artifact — those histograms step ×3 from
0.66 s to 1.97 s, so any tail interpolates to "~1.93" identically across streams; the
per-query buckets should be aligned with the phase buckets in a follow-up.) The state-change build round above then graded on the rig (16-minute window, same env,
+54%-load producer split active): process p99 1.380 s → 0.992 s — the first stage under the
1 s contract bar — with persist p99 1.483 → 1.367 and throughput up ~3% at 18,429 tx/s. The
profile confirms the mechanism: the builder allocation family is absent entirely, the
state-change subtree fell from 23.1% to 17.4% of allocated bytes, and resident heap dropped
~0.9 GB (scanobject 27.1% → 24.8%). A separate GOGC A/B on the pre-round image found
GOGC=100 reproduces the same tail win via pacing alone (1.380 → 0.992, +2.7% tx/s), so the
rig's deployment config ships GOGC=100 as well.
The pod defaults changed: DB pool default rises to 12 connections (persist holds 8 at the commit barrier — five COPY siblings, the balances and trustlines siblings, and the coordinator).
Deploy notes
state_changesindex changes (see above; done on loadtest, pending on dev mainnet/testnet/prod together with the earliertransactions_ledger_created_at_idxrestore, which prod still needs).--live-persist-max-batch-size(envLIVE_PERSIST_MAX_BATCH_SIZE), default 1 (commit-per-ledger, unchanged behavior). Raising it costs memory: the rotation holds 2N+1 buffers, each retaining a merged ledger's maps. The loadtest rig sets 3, which is what its ingest container fits alongsideGOMEMLIMIT.LOADTEST_META_PIPE_PATHS→LOADTEST_META_SOURCEStogether with this image (the kube-side topology split ships both; between image roll and manifest sync the rig crash-loops on the missing flag, which the scheduled reset absorbs).histogram_quantile(0.99, sum by (le, phase) (rate(wallet_ingestion_phase_duration_seconds_bucket[15m])))and awallet_ingestion_persist_batch_sizep50 panel.