Fix the DB query-duration buckets, the oldest-ledger index path, and compression scheduling - #686
Open
aditya1702 wants to merge 62 commits into
Open
Fix the DB query-duration buckets, the oldest-ledger index path, and compression scheduling#686aditya1702 wants to merge 62 commits into
aditya1702 wants to merge 62 commits into
Conversation
This was referenced Aug 13, 2026
aditya1702
force-pushed
the
blend/pr6-integration-tests
branch
2 times, most recently
from
August 17, 2026 19:04
c7561b5 to
996e7a2
Compare
An error occurred while trying to automatically change base from
blend/pr6-integration-tests
to
blend/pr5-graphql
August 17, 2026 20:47
…flag ingestService carried four fields (appTracker, getLedgersLimit, knownContractIDs, contractMetadataService) and Indexer two (ingestionMetrics, networkPassphrase) that were assigned at construction and never read — consumers get these through their own constructors or through internal/ingest.Configs. Deleting them also drops AppTracker, GetLedgersLimit, and ContractMetadataService from IngestServiceConfig (the checkpoint service keeps its own metadata-service wiring). Also removes the --latest-ledger-cursor-name flag, which only fed a deprecation warning, and the phantom load_current_state phase label from the ProtocolStateProcessingDuration help text (never emitted).
Every remaining parameter of persistLedgerData had exactly one production value: cursorName was always the latest-ledger cursor (the blind-Update branch was reachable only from tests), ledgerMeta was never nil, and the numTxs/numOps returns only fed two counters the buffer already knows via GetNumberOfTransactions/GetNumberOfOperations. The signature is now (ctx, ledgerSeq, ledgerMeta, plan, contractData, buffer) error, the cursor write is unconditionally guarded, and the contractDataMemo nil-receiver branch (test-only) is gone. IndexerBufferInterface shrinks to its real seam — the indexer's fold-in plus the five getters insertIntoDB reads; everything else is reached on the concrete *IndexerBuffer. Tests now exercise the guarded cursor path persistLedgerData actually runs in production.
…Backoff persistLedgerDataWithRetry (né ingestProcessedDataWithRetry) hand-rolled the same attempt loop, exponential backoff, cap, and context handling that utils.RetryWithBackoff already provides — the body is now a single RetryWithBackoff call with isPermanentPersistError as the classifier. Metric semantics preserved: transient retries count RetriesTotal, permanent errors count ErrorsTotal and return immediately, exhaustion counts both plus RetryExhaustionsTotal, and context cancellation counts neither. RetryWithBackoff's exhaustion return now wraps the exported sentinel utils.ErrRetriesExhausted alongside the final error, which is how the wrapper tells exhaustion apart from a permanent-error exit.
…ist stages The live loop ran fetch (~0.3s of waiting), process_ledger (~0.8s of Go CPU), and the DB persist (~1.1s) strictly in sequence, so the per-ledger time was their sum — above a 2s ledger cadence at high transaction volume even though each stage individually fits. The loop is now a three-stage errgroup pipeline over depth-1 channels: while ledger N persists, N+1 processes and N+2 is fetched, making the ledger time the slowest stage instead of the sum. Persist stays strictly sequential in ledger order (the guarded cursor and per-protocol CAS chain advance N-1 → N) and is the only writing stage, so the advisory-lock liveness probe moves there. Classification planning also stays on the persist stage: its known-hash lookup is a non-transactional pool read that must observe the previous ledger's committed protocol_wasms rows. Two IndexerBuffers rotate through a freeBuffers channel, cleared only at take so a clear can never tear a persist still reading the maps the getters alias. Any stage error cancels the pipeline and the process exits, resuming from the cursor on restart — the Duration metric now sums the ledger's stage times, since wall time under a pipeline would count queue wait.
…with coordinated late commits The live persist ran its three COPY families — transactions(+accounts), operations(+accounts), state_changes — sequentially inside one transaction on one connection, so the insert phase cost their sum and a single Postgres backend did all the index maintenance. They now stream concurrently on three sibling connections, each in its own transaction, while the coordinating transaction stages everything else (assets, contracts, classification, protocol state, token changes, cursor). The table sets are disjoint with no FKs among them, and every goroutine only reads the quiescent buffer. Commits are the visibility point and are held until all four transactions have done their work: siblings commit first (sub-ms each), the coordinating transaction — whose cursor is the authority on which ledgers exist — strictly last. A failure before the first commit rolls everything back and stays retryable exactly as before; a failure after it wraps the new ErrPartialPersist sentinel, which isPermanentPersistError classifies as fatal, because COPY has no ON CONFLICT and re-running the ledger would collide on primary keys. The only crash state this ordering can produce is orphaned bulk rows for the single ledger past the committed cursor, so startup runs IngestStoreModel.DeleteRowsAboveLedger before resuming: one transaction of TOID-bounded deletes (rows of ledgers > cursor are exactly rows with TOID >= toid.New(cursor+1,0,0)) across the five bulk tables, kept chunk-local by each table's chunk skipping. Backfill keeps the single-transaction insertIntoDB path unchanged.
…ommits When the process stage finishes ledgers faster than persist drains them, the persist stage now folds up to --live-persist-max-batch-size (default 5) consecutive ledgers into one commit set: each sibling COPY streams the whole batch on its connection, the coordinating transaction stages the batch's ledgers in order, and the commit barrier fires once — amortizing COPY setup, index-page churn, and fsyncs across the backlog. While the pipeline keeps pace every batch has size 1 and behavior is unchanged. A ledger with classification inputs always opens its own batch: its plan's pool reads see exactly what the previous batch committed, which preserves the deployed-contract-sees-prior-wasm invariant. The cursor stays the authority — one guarded update lands on the batch's last ledger, and a pre-commit failure rolls back and retries the whole batch. wallet_ingestion_persist_batch_size observes coalescing; per-ledger duration histograms record each ledger's amortized share so panels stay comparable across batch sizes.
…on WAL flush SET LOCAL synchronous_commit = off on each sibling session removes up to three serialized fsync waits per persist. Durability is unchanged: the coordinating transaction commits synchronously and strictly last, and its flush covers all earlier WAL including the sibling commit records, so a durable cursor implies durable siblings; rows a crash could lose are exactly the unacknowledged ones startup reconciliation deletes.
The classification-safety cut isolated any ledger carrying protocol wasm/contract observations, but a plan's pool reads are only unsound for inputs no committed batch has classified yet — and re-observations of known contracts are the overwhelmingly common case (synthetic loadtest traffic re-observes the same token contracts every ledger, which cut every batch to size one and disabled batching entirely; goroutine dumps showed the process stage starved of buffers held by the always-cut pending queue). The persist goroutine now keeps seen-sets of wasm hashes and contract IDs folded in after each successful commit; only a ledger introducing unseen inputs opens its own batch. A rolled-back batch marks nothing, and a restart starts empty — conservative until re-warmed.
…ary key TOID encoding makes a state change's parent transaction to_id derivable from its operation_id (to_id = operation_id &^ 0xFFF), so BatchGetByOperationID and BatchGetByOperationIDs now bind (ledger_created_at, to_id, operation_id) — a primary-key prefix seek — instead of filtering operation_id across every entry sharing the timestamp (measured 7 vs 307 buffers on a single-chunk fixture). That leaves idx_state_changes_operation_id with no consumer; it is removed. idx_state_changes_account_category narrows to idx_state_changes_account_id (account_id + the PK sort key): the category and reason columns only ever seeded an ordered seek for the one filter shape that pins both, while every other shape and BatchGetAccountStateChangesByToIDs seek on account_id alone. Filtered variants scan-and-filter the account's rows on the active chunk and prune compressed chunks via the existing bloom sparse indexes. Two fewer btree columns per insert on the highest-volume table. Already-migrated DBs need the one-time manual DDL (drop both old indexes, create the narrowed one); TimescaleDB rejects CREATE INDEX CONCURRENTLY on hypertables.
…lings The live persist path's sibling COPY streams grow from three to five: transactions_accounts and operations_accounts — the two largest index-maintenance payloads, each carrying an account_id-leading unique PK — stream concurrently with their parent tables instead of serially behind them. TransactionModel and OperationModel each split BatchCopy into a parent-table COPY and BatchCopyAccounts for the link table, with the link COPY's duration and batch size recorded under its own metric labels rather than folded into the parent's. Each sibling now writes exactly one table, so the disjointness at the commit barrier is per-table; the backfill path calls the five inserts in the old order and is behaviorally unchanged. Persist holds six connections at its commit barrier; the pool default rises to 12 so the persist path never queues on Acquire behind the advisory-lock session or pool-side classification reads.
… random order The native_balances and trustline_balances batch upserts arrive in Go map-iteration order and unconditionally rewrite every conflicting row. On the loadtest rig the pattern measured 60+ heap buffer touches per row (vs ~6 healthy) against a table bloated to 97% free space by the churn. Two changes: rows sort by the primary-key columns before the UNNEST arrays are built, so the btree descent and heap touches run in key order; and the DO UPDATE carries an IS DISTINCT FROM guard, so an identical row produces no new tuple version, no dead tuple, no WAL, and no index churn.
…cores runtime.NumCPU() reports the node's cores in a container with a CPU limit, so a 6-core pod on a 36-core node ran 72 indexer workers over 6 schedulable threads — pure scheduler churn on CPU-bound work. GOMAXPROCS honors the limit.
The buffer stored transactions keyed by hash while tracking their participants keyed by ToID. The streaming-loadtest backend's merged bootstrap ledgers can carry the same envelope at several tx-set positions — distinct ToIDs, one hash — and the two maps then disagreed: duplicate transactions were silently dropped from the transactions COPY while their participant links survived, whose ledger_created_at lookup missed and COPYed the zero timestamp into a year-0001 chunk (observed on the rig: 6 orphaned transactions_accounts rows, 5 missing transaction rows across bootstrap ledgers). Real networks cannot repeat a hash within or across ledgers, so both maps now share the ToID key domain by construction, mirroring the operations pair. BatchCopyAccounts on both link tables now errors on a ToID/opID with no parent row instead of silently writing a zero timestamp.
Batching is opt-in per deployment: at mainnet/testnet cadence the close time comfortably exceeds the persist time, so every ledger commits on its own; high-TPL/short-block deployments raise the flag to amortize backlogs.
…dger's close time DeleteRowsAboveLedger scanned every chunk TOID chunk-skipping stats could not exclude — stats only cover compressed chunks, and three of the five tables have no TOID-leading index — which took over 10 minutes on a loadtest DB carrying hours of full ledgers, long enough that the unread meta pipes wedged all three generators into liveness-probe kills. Close times are monotone, so every orphan above the cursor carries ledger_created_at at or after the cursor ledger's own close time; binding that as a partition-column predicate statically excludes all older chunks regardless of compression, and turns state_changes' scan into a primary-key range seek. A cursor ledger with no transactions row leaves the bound unresolvable and the deletes fall back to the unbounded scan.
…eachable A buffer stays checked out from the moment the process stage takes it until the batch carrying it commits, so the rotation needs 2*cap buffers to sustain a full batch: cap for the batch in flight, cap-1 queued in processed, one being filled. It held cap+1, so process starved on freeBuffers after filling (cap+1)-k and the batch settled at the fixed point k = (cap+1)-k. The batch size was therefore pinned below the configured cap at every load — loadtest measured exactly 3.00 against a cap of 5 across 17 sample windows spanning 3.9k to 9.8k tx/s, and the new test measures 2.04 against a cap of 3 when the pool is sized the old way.
…e bar The phase histogram stepped 0.5 -> 1 -> 2 seconds, so a stage sitting at 0.9s and a stage sitting at 1.9s both reported the same p99 bucket. The pipeline's contract is that the slowest stage's p99 stays under the ledger close time, so the metric could not grade the thing it exists to grade. Boundaries now land exactly on 0.6s (Phase-3 block rate) and 1s (the loadtest rig's 1.67x-sized merged ledgers), with 0.5 and 0.75 around them.
…ling The native-balance and liquidity-pool upserts ran serially inside the coordinating transaction, lengthening its critical path while the COPY siblings streamed concurrently. Their tables carry no foreign keys, so a sixth sibling now carries them; the trustline and SAC balances stay on the coordinating transaction because their parent rows (trustline_assets, contract_tokens) are staged uncommitted there, and a sibling committing first would fail FK checks against committed state. Reconciliation (DeleteRowsAboveLedger) does not cover the balance tables: a crash can leave balance rows above the committed cursor, which is safe because the upserts are idempotent and reapply when those ledgers re-ingest.
A pool-share balance is read alongside its pool's reserves, and ingestion writes both tables in one transaction with the pool row first, so a balance whose pool is missing can only be an ingestion bug. The new foreign key turns that bug into a loud write-time failure, matching the trustline_balances -> trustline_assets constraint; deferred so the commit-time check covers same-transaction pool deletions. Verified orphan-free on the dev and loadtest databases before adding.
…bling Each balance family now rides the transaction that stages its foreign-key parents: trustline_assets and trustline_balances move from the coordinating transaction onto a new sibling, alongside the balances sibling that carries pools and pool-share balances under the new pool foreign key. Every FK is checked at its own transaction's commit, and the coordinator's serial path shrinks to contracts, classification, protocol state, SAC balances, and the cursor. SAC balances stay coordinated: their parent (contract_tokens) is also written by the classification path there, and a same-key insert from two concurrent transactions could deadlock at the commit barrier.
… the pool BatchGetByContractIDs queried the pool, so a mid-batch ledger's membership lookup could not see protocol_contracts rows staged earlier in the same batch's uncommitted coordinating transaction. A contract deployed and classified at the batch head, then invoked mid-batch without touching its instance entry, buffered nothing on the later ledger — its events were silently dropped for event-only processors (SEP-41; Blend re-reads through the transaction and was unaffected). Unreachable at the default batch size of 1; the loadtest rig runs 3. The lookup now takes a db.Querier and live ingestion passes the coordinating transaction, mirroring GetByProtocolID.
…mo semantics BatchCopyAccounts on transactions and operations returned a link-row count every caller discarded — the count is already observed into the BatchSize metric inside the method — so both now return plain error, matching the balance models' BatchCopy shape. contractDataMemo's contract gets a unit test: get() always yields a rangeable non-nil map, and the extraction walk runs at most once no matter how many retry attempts share the memo. The startup-reconciliation comment now says a crash orphans at most the persist batch past the cursor; batching had outgrown "the single ledger".
Protocol history rows are state_changes rows, so PersistHistory on the coordinating transaction had two of our own transactions inserting into the same hypertable concurrently. At a chunk boundary TimescaleDB serializes chunk creation and the waiter blocks on the other transaction's end — but the coordinator only commits after every sibling goroutine returns, so if the sibling is the waiter the pipeline hangs forever, invisible to Postgres's deadlock detector. All state_changes writes now go through the one state_changes sibling transaction, serialized by a mutex (pgx.Tx is not concurrency-safe). The CAS stays on the coordinator: siblings commit strictly first, so a committed cursor still implies committed history rows, and rows stranded above the cursor by a crash are already removed by DeleteRowsAboveLedger.
…wrapper memos The persist stage extracted each ledger's ContractData changes through tx.GetChanges(), which rebuilds and ledger-key-sorts every operation's change group — work the process stage had already done and memoized on the shared TransactionOperationWrapper. On the loadtest rig that duplicate path was 8.7s of a 239.7s profile (3.6% of ingest CPU, plus its allocation/GC halo), all of it on the serial persist goroutine. processTransaction now collects each successful transaction's ContractData changes from the wrappers' memoized Changes() slices, mirroring GetChanges' composition per meta version (tx-level before, operations ascending, tx-level after); transaction-level segments are only materialized when a type-tag scan of the raw group mentions ContractData at all. Results fold into the ledger buffer in transaction order and reach the persist stage on the buffer it already carries, replacing the contractDataMemo and the transactions plumbing through processedLedger; ProcessLedger no longer returns the materialized transactions. Merge gate: TestProcessLedgerTransactions_ContractDataChangesMatchReaderExtraction compares the buffer's collection against the reader-based tx.GetChanges() reference over every real-ledger fixture; synthetic tests pin the success gate, the meta-version guard, wrapper-memo serving, and the no-wrapper fallback.
AddressByteaMemo caches the 33-byte form per unique address string within one COPY build, where addresses repeat heavily (a token's transfers share token_id, an account's rows share account_id). A hit skips the strkey decode and the 33-byte allocation entirely: 278ns/200B/4allocs → 9.3ns/0B/0allocs. AddressBytea.Value now rejects payloads that are not exactly 32 bytes instead of silently truncating them (an M-address decodes to 40 bytes; unreachable from current callers, defensive).
dedupeParticipants keyed a map on id.Address(), strkey-encoding every participant just to throw the string away — the caller immediately re-encodes the survivors. It now compares the raw ed25519 keys with a linear scan over the kept prefix, filtering in place: zero allocations, zero encodes, and the output keeps deterministic first-seen order instead of map order (downstream folds into a set, so ordering was and stays unobservable). The replaced implementation stays in the test file as a differential oracle.
GetOperationsParticipants hoists the envelope operations and ledger close time out of the per-op loop, sizes its result map, and drops the unreachable merge branch (opID embeds the operation index, so each iteration writes a distinct key). Per-op participant sets are pre-sized. The metric-only unique-participants map is gone: the tx-level participant set doubles as the cardinality accumulator once its slice snapshot is taken.
…ator The Soroban participant walk allocated a set per processor, per auth entry, and per invocation-tree node, merging each level with Union (which allocates a third set per call), and re-encoded the op source address in every sub-processor. The whole path now adds into the single set GetOperationParticipants already built: no intermediate sets, no Unions, one source encode per op. Riding along, profile-guided: - The per-node reflective utils.IsEmpty guard (which heap-escapes the whole invocation) is a typed nil-arm check: a zero-value invocation reports the ContractFn arm with a nil pointer, which GetContractFn would dereference. - Sub-processors fetch MustInvokeHostFunctionOp once for both the host-function args and the auth entries (was two to three union-arm copies per op). - Invocation nodes and auth entries are visited by pointer instead of by value. Behavior is pinned by the existing exact-set expectations plus new tests for the zero-value invocation and the muxed-source M-address form.
Every state_changes row decoded up to six strkey addresses from scratch, and the values repeat heavily within one batch: every transfer of a token shares its token_id and an account's fee/balance/effect rows share its account_id. One AddressByteaMemo per BatchCopy call now serves all six columns, replacing the pgtypeBytesFromNullAddressBytea helper whose only caller this was.
Participants are deduplicated per transaction/per operation upstream, so a busy account is decoded once per tx or op it touches in transactions_accounts and operations_accounts. One AddressByteaMemo per BatchCopyAccounts call collapses that to one decode per unique address per batch.
SourceAccount() returned a pointer, heap-escaping a stack local on every call in the common no-explicit-source branch — and it is called several times per operation across the participants and effects paths. It now returns the xdr.MuxedAccount by value; callers that need a pointer or addressability take a local first.
TransactionOperationWrapper copied the whole ingest.LedgerTransaction (envelope, result, meta, fee changes, ledger header) into every per-operation wrapper. The field is now a pointer: GetOperationsParticipants materializes the transaction once and every wrapper of the ledger transaction aliases it. The wrapper path is read-only, and each per-transaction worker owns its transaction, so nothing is shared across goroutines. makeFeeBumpOp (test util) clones the transaction before rewriting the envelope so a fee-bump variant cannot write through the base op's now-shared pointer.
AssignStateChangeOrdinals runs four times per transaction, and its group counter map — sized to the stream — was allocated and discarded on every call. Streams are a handful of changes, so counting earlier same-group elements with a quadratic scan is cheaper and allocation-free, and produces byte-identical IDs.
…nge path - Sponsorship scans allocate their detail maps only after detecting a transition; accounts with no sponsored signers skip the SDK's SponsorPerSigner map-and-slice build entirely. - EffectOutput drops five write-only fields (EffectID, EffectIndex, LedgerClosed, LedgerSequence, AddressMuxed) and the per-effect fmt.Sprintf that populated them; nothing reads them. - Transaction hashes are hex-encoded at the rare log/error sites instead of eagerly per operation. - Soroban-only processors check the operation type before installing their duration-metric closure, so classic operations pay neither the timestamps nor the label lookup; invalid-op-type calls no longer land in the histogram. - ContractDeploy's dedupe map allocates on first deploy, not per call. - Zero-effect operations return before the master builder is built.
Deriving a SAC contract ID costs a SHA-256 over the asset's contract-ID preimage plus a strkey encode, and both the effects and token-transfer processors recompute it for the same few assets on every event. Each processor now memoizes by asset identity; entries are content-derived under the processor's fixed network passphrase, so they never invalidate, and sync.Map keeps the lookup safe across the indexer's pool workers.
The sponsorship effect paths had no coverage: no test exercised addSignerSponsorshipEffects, addLedgerEntrySponsorshipEffects, or the accountHasSponsoredSigner guard that spares the SponsorPerSigner build for accounts sponsoring nothing. Covers every transition (created, removed, updated, unchanged, whole-entry removal), the guard's over-admit-never-over-reject contract, deterministic signer ordering, and each sponsorable entry type's details shape.
Every NewStateChangeBuilder and Clone() heap-allocated a ~480-byte builder, and the hot paths clone constantly — a plain payment paid four heap builders for three state changes. The builder is now used by value: With* methods mutate their own copy and return it, branching is plain assignment, Clone() is gone, and builders stay on the stack. The write-only metricsService field is dropped from the builder along with the constructor parameter. parseTrustline now returns the builder it produced so the balance-authorization change emitted for a new trustline keeps inheriting the resolved token and the trustline limit exactly as it did through the shared pointer; whether that inherited limit belongs on a BALANCE_AUTHORIZATION row is flagged in the PR as a follow-up decision. Helper signatures across the effects, token-transfer, contract-deploy, SAC, SEP-41, and Blend processors take the builder by value; output is byte-identical everywhere.
The AddressMuxed removal left github.com/guregu/null unused module-wide, and the sponsorship fixtures always build entries for sponsoredAccount, so the account parameter comes off both helpers (unparam).
…tion XDR encoder The signers-only skip asked the SDK's AccountChangedExceptSigners, which deep-copies both account entries, XDR-marshals each, and compares the bytes — 2.6% of ingest CPU and 5.7% of allocated bytes at ~18k tx/s. A local field comparison preserves the exact semantics (only the Signers slice is ignored; NumSubEntries and V2 signer-sponsoring IDs still count; a missing extension equals V1 with zero liabilities), pinned by a 25-case differential test against the SDK function as oracle. The always-nil error plumbing collapses with it. ConvertOperation's per-operation XDR marshal allocated a fresh encoder and growth-doubled buffer each call; a pooled EncodingBuffer reuses the scratch across pool workers and allocates only the retained, exact-size copy of each operation's bytes.
The quadratic scan's justifying comment — streams are a handful of changes per transaction — is true for the indexer's per-transaction call but false for the sep41 and blend processors, which assign ordinals over a whole staged migration window at persist time, inside the CAS-guarded window transaction, with nothing bounding the slice but the window. A counting map keeps the assignment linear at every call site; IDs are byte-identical to the scan's. The per-call map allocation is noise on the live path: four calls per transaction against a process stage that allocates five orders of magnitude more per ledger.
…s into functions CreateContractV1OpProcessor, CreateContractV2OpProcessor, and InvokeContractOpProcessor each held one field, had one method, and were constructed and consumed on adjacent lines inside participantsForSorobanOp's switch — with no other consumer in the tree. Each method also re-checked the op type and host-function type the switch had already decided, and re-copied the op body. They are now unexported functions taking the network passphrase and the InvokeHostFunctionOp the caller already holds.
…n table missed Three accountChangedExceptSigners branches had no reaching table row in TestAccountChangedExceptSigners_MatchesSDK: a differing AccountId, an InflationDest transitioning set to nil, and sponsoring-ID slices of different length (nil vs one element). Each new row differs from the base entry in exactly one field, so its verdict can only come from that field's guard; the SDK oracle agrees on all three.
Only Soroban host functions write ContractData entries, so transactionContractDataChanges now returns before walking or allocating anything when the transaction carries no SorobanTransactionData — the common case on a payment-dominated ledger. The per-transaction wrapper lookup becomes a dense slice instead of a map, and the result map is allocated on the first change actually collected instead of eagerly. The reader-extraction equivalence test still passes over the real-ledger fixtures, which pins that the gate drops nothing a Soroban transaction produces.
…m one memo AssetContractIDMemo is the package's single cache for the SHA-256-plus- strkey derivation: token_transfer's hand-inlined sync.Map becomes a memo field, and the two SAC comparison sites that recomputed the derivation per event (isSACContract, trustlineEntryMatches) now read through a memo on the processor. Three accessors normalize the three caller shapes onto one complete key, so the same asset lands on one entry no matter which path reaches it; the key's completeness and the accessor split's cost reasoning are documented on the type. Trustline matching compares contract IDs strkey-encoded — equivalent by injectivity — which retires a per-call decode. The preimage path in contract_operations stays uncached: it fires roughly once per asset lifetime (SAC deploys), not per event.
aditya1702
force-pushed
the
ingest-db-observability
branch
from
August 22, 2026 13:37
58636ea to
65a2623
Compare
aditya1702
changed the base branch from
blend/pr6-integration-tests
to
live-ingest-processor-perf
August 22, 2026 13:40
The wallet_db_query_duration_seconds histogram topped out at 0.38s (ExponentialBuckets(0.0001, 2.5, 10)), so every multi-second bulk COPY landed in +Inf and histogram_quantile panels clipped at ~0.38s, under-reporting exactly the queries being optimized. Widen to ExponentialBuckets(0.0001, 3, 12) — 0.1ms through ~17.7s.
…-ledger lookup The oldest-ledger lookup (SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1) has two consumers — backfill gap detection's left bound (GetOldestLedger) and the hourly reconcile_oldest_cursor TimescaleDB job — and neither had an ordered path: the transactions migration dropped TimescaleDB's default partition-column index as consumer-less, an audit that counted WHERE-clause consumers and missed that this query consumes the index's ordering. Without it the planner pulls the first row from every chunk instead of running an ordered ChunkAppend that stops at the oldest (12-187s on an 87GB DB; the hourly job alone accounted for 45% of all DB disk-read time and flushed shared_buffers every run). Keep the default index (fresh databases get it from create_hypertable; already-migrated environments need a one-time manual `CREATE INDEX transactions_ledger_created_at_idx ON transactions (ledger_created_at DESC)`) and drop the to_id tie-break from both query sites: close times increase strictly, so rows sharing a ledger_created_at carry the same ledger_number and the tie-break could only force an incremental sort on top of the index's pathkeys. Covered by a new test that runs the reconciliation job via run_job against out-of-order chunks.
…the schedule interval Each hypertable's columnstore policy is auto-created with an identical schedule, so all five fire at the same instant and compress their just-closed chunks concurrently — an I/O storm that starves the persist stage (measured on the loadtest rig: persist p50 0.68s -> 3.0s for 18 of every 60 minutes). Converge each policy onto a fixed schedule anchored to the interval grid with a distinct per-table slot offset, so at most one policy comes due at a time. Jobs already on their slot are left untouched across restarts, preserving next_start and run history.
…uration Per-commit observations made the series incomparable with process_ledger: counts tracked commits, not ledgers, and the p99 read as 'commit latency' where every other phase reads per-ledger. Each ledger now observes the FULL wall time of the commit that carried it — still never divided by batch size, so slow batched commits stay visible in the ledger-close-time grading buckets, weighted by how many ledgers they delayed.
aditya1702
force-pushed
the
ingest-db-observability
branch
from
August 25, 2026 18:49
65a2623 to
26b73ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four independent fixes, one commit each — reviewable in any order. Review time: ~20 min. One manual-DDL warning below.
Found while profiling live ingestion: a metric that clipped exactly the queries being optimized, a missing index that made an hourly job 45% of all DB disk reads, five compression policies firing at the same instant, and a phase histogram whose counts stopped matching its siblings.
1. Query-duration buckets clipped at 0.38 s
wallet_db_query_duration_secondstopped out at 0.38 s: every multi-second bulk COPY landed in+Infand p99 panels flattened. Widened to 0.1 ms – ~17.7 s.2. Oldest-ledger lookup had no ordered path
SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1(backfill gap detection + the hourlyreconcile_oldest_cursorjob) had no index to walk — an earlier audit dropped TimescaleDB's default partition-column index after counting only WHERE-clause consumers, missing that this query consumes the index's ordering. The planner pulled the first row from every chunk: 12–187 s on an 87 GB DB; the hourly job was 45% of all DB disk-read time.Fix: keep the default index; drop the
to_idtie-break from both query sites (close times increase strictly — the tie-break could only force an extra sort). Tested by running the reconciliation job against out-of-order chunks.Warning
Manual DDL on already-migrated databases (fresh DBs get the index from
create_hypertable). Applied on the loadtest DB; pending on dev/staging/prod:Note
2025-06-10.2-transactions.sql(already shipped) is edited in place, not superseded. Deliberate: wallet-backend is pre-release, and the baseline files stay the single readable statement of each table's final shape.3. Compression policies all fired at once
All five hypertables' columnstore policies were auto-created on identical schedules and compressed their just-closed chunks concurrently — an I/O storm that took persist p50 from 0.68 s to 3.0 s for 18 of every 60 minutes on the rig. Each policy is now anchored to a distinct slot offset on the schedule grid, so at most one fires at a time; jobs already on their slot are untouched across restarts.
4. insert_into_db now observes once per ledger, at full commit duration
The phase histogram recorded one observation per persist commit, so under batching its counts tracked commits while every other phase tracked ledgers — panels comparing phases (or grading p99 against ledger close time) mixed units.
Each ledger now observes the full, undivided wall time of the commit that carried it:
process_ledger;persist_batch_sizestill carries the coalescing alongside.Stacked: #679 → #684 → #685 → this → #699. Combined rig result: 18,458 tx/s, process p99 0.992 s; PRs not separately re-benchmarked.