Cut process-stage CPU and allocations across the indexer and processors - #685
Open
aditya1702 wants to merge 24 commits into
Open
Cut process-stage CPU and allocations across the indexer and processors#685aditya1702 wants to merge 24 commits into
aditya1702 wants to merge 24 commits into
Conversation
This was referenced Aug 13, 2026
Open
aditya1702
force-pushed
the
live-ingest-persist-path
branch
from
August 20, 2026 19:35
e123d77 to
d5acde6
Compare
aditya1702
force-pushed
the
live-ingest-processor-perf
branch
3 times, most recently
from
August 22, 2026 02:51
c8f21d9 to
c4aa67e
Compare
…per processor Every state-change and effects processor independently called Transaction.GetOperationChanges for the same operation, and the SDK accessor re-decodes the op's meta, re-allocates, and re-sorts the changes on every call — 19% of process-stage CPU in GetChangesFromLedgerEntryChanges, 14% in sortChanges, and a large share of GC pressure on a production profile. All of an operation's processors share one TransactionOperationWrapper, so Changes() now memoizes the extraction on the wrapper; all 14 call sites route through it, audited read-only (the memoized slice is shared — callers must not mutate it or write through Pre/Post). A wrapper copied to describe a different operation resets its memo so the cache always matches Index. BenchmarkProcessRealLedger on the five real-ledger fixtures: 40-50% less wall time, ~50% fewer bytes and ~57% fewer allocations per ledger.
Pairing metas with envelopes needs every envelope's hash, and building it was the pipeline's largest serial stretch: on the loadtest rig it ran ~0.37s single-threaded ahead of the fan-out, 41% of the process stage, of which 85% was the hashing itself. Reading transactions out of a LedgerCloseMeta now hashes envelopes across the indexer's worker pool instead of on the caller's goroutine. Three costs went with it. The network id is a hash of the passphrase alone, so it is computed once per ledger rather than once per transaction. Each chunk marshals through one xdr.EncodingBuffer, which reuses its scratch space across transactions instead of allocating a buffer per hash — this path alone was 8.5% of everything the ingest process allocated. And the envelope's transaction is tagged in place rather than copied by value. On the real pubnet fixtures, where the fan-out barely engages at a few hundred transactions per ledger: 2.8-3.8x faster, 4.3-8.6x fewer allocations. Also restores the independence of the two reader-based oracles in the test suite. They materialized their transactions through the same function they were the oracle for, so they now go through getLedgerTransactionsViaReader, which drives the SDK reader and shares no code with the fan-out. That helper is the merge gate for this change too, via TestGetLedgerTransactions_EquivalenceOnRealLedgers.
isLiquidityPool and isClaimableBalance ran a full strkey decode — base32, CRC16 validation, and a byte copy of the input — on both endpoints of every transfer event. Those endpoints are ordinary account and contract addresses almost every time, so nearly all of that work only ever concluded "no". A strkey version byte is a multiple of 8, so its top five bits are exactly what the first base32 character encodes: only an 'L' can decode to a liquidity pool and only a 'B' to a claimable balance, whatever the payload. Checking that character first skips the decode for everything else, and a string that passes it still goes through the decode, so validation is unchanged. Together this was 13.6% of the token-transfer processor on the loadtest rig.
…wrapper memos The persist stage extracted each ledger's ContractData changes through tx.GetChanges(), which rebuilds and ledger-key-sorts every operation's change group — work the process stage had already done and memoized on the shared TransactionOperationWrapper. On the loadtest rig that duplicate path was 8.7s of a 239.7s profile (3.6% of ingest CPU, plus its allocation/GC halo), all of it on the serial persist goroutine. processTransaction now collects each successful transaction's ContractData changes from the wrappers' memoized Changes() slices, mirroring GetChanges' composition per meta version (tx-level before, operations ascending, tx-level after); transaction-level segments are only materialized when a type-tag scan of the raw group mentions ContractData at all. Results fold into the ledger buffer in transaction order and reach the persist stage on the buffer it already carries, replacing the contractDataMemo and the transactions plumbing through processedLedger; ProcessLedger no longer returns the materialized transactions. Merge gate: TestProcessLedgerTransactions_ContractDataChangesMatchReaderExtraction compares the buffer's collection against the reader-based tx.GetChanges() reference over every real-ledger fixture; synthetic tests pin the success gate, the meta-version guard, wrapper-memo serving, and the no-wrapper fallback.
AddressByteaMemo caches the 33-byte form per unique address string within one COPY build, where addresses repeat heavily (a token's transfers share token_id, an account's rows share account_id). A hit skips the strkey decode and the 33-byte allocation entirely: 278ns/200B/4allocs → 9.3ns/0B/0allocs. AddressBytea.Value now rejects payloads that are not exactly 32 bytes instead of silently truncating them (an M-address decodes to 40 bytes; unreachable from current callers, defensive).
dedupeParticipants keyed a map on id.Address(), strkey-encoding every participant just to throw the string away — the caller immediately re-encodes the survivors. It now compares the raw ed25519 keys with a linear scan over the kept prefix, filtering in place: zero allocations, zero encodes, and the output keeps deterministic first-seen order instead of map order (downstream folds into a set, so ordering was and stays unobservable). The replaced implementation stays in the test file as a differential oracle.
GetOperationsParticipants hoists the envelope operations and ledger close time out of the per-op loop, sizes its result map, and drops the unreachable merge branch (opID embeds the operation index, so each iteration writes a distinct key). Per-op participant sets are pre-sized. The metric-only unique-participants map is gone: the tx-level participant set doubles as the cardinality accumulator once its slice snapshot is taken.
…ator The Soroban participant walk allocated a set per processor, per auth entry, and per invocation-tree node, merging each level with Union (which allocates a third set per call), and re-encoded the op source address in every sub-processor. The whole path now adds into the single set GetOperationParticipants already built: no intermediate sets, no Unions, one source encode per op. Riding along, profile-guided: - The per-node reflective utils.IsEmpty guard (which heap-escapes the whole invocation) is a typed nil-arm check: a zero-value invocation reports the ContractFn arm with a nil pointer, which GetContractFn would dereference. - Sub-processors fetch MustInvokeHostFunctionOp once for both the host-function args and the auth entries (was two to three union-arm copies per op). - Invocation nodes and auth entries are visited by pointer instead of by value. Behavior is pinned by the existing exact-set expectations plus new tests for the zero-value invocation and the muxed-source M-address form.
Every state_changes row decoded up to six strkey addresses from scratch, and the values repeat heavily within one batch: every transfer of a token shares its token_id and an account's fee/balance/effect rows share its account_id. One AddressByteaMemo per BatchCopy call now serves all six columns, replacing the pgtypeBytesFromNullAddressBytea helper whose only caller this was.
Participants are deduplicated per transaction/per operation upstream, so a busy account is decoded once per tx or op it touches in transactions_accounts and operations_accounts. One AddressByteaMemo per BatchCopyAccounts call collapses that to one decode per unique address per batch.
SourceAccount() returned a pointer, heap-escaping a stack local on every call in the common no-explicit-source branch — and it is called several times per operation across the participants and effects paths. It now returns the xdr.MuxedAccount by value; callers that need a pointer or addressability take a local first.
TransactionOperationWrapper copied the whole ingest.LedgerTransaction (envelope, result, meta, fee changes, ledger header) into every per-operation wrapper. The field is now a pointer: GetOperationsParticipants materializes the transaction once and every wrapper of the ledger transaction aliases it. The wrapper path is read-only, and each per-transaction worker owns its transaction, so nothing is shared across goroutines. makeFeeBumpOp (test util) clones the transaction before rewriting the envelope so a fee-bump variant cannot write through the base op's now-shared pointer.
AssignStateChangeOrdinals runs four times per transaction, and its group counter map — sized to the stream — was allocated and discarded on every call. Streams are a handful of changes, so counting earlier same-group elements with a quadratic scan is cheaper and allocation-free, and produces byte-identical IDs.
…nge path - Sponsorship scans allocate their detail maps only after detecting a transition; accounts with no sponsored signers skip the SDK's SponsorPerSigner map-and-slice build entirely. - EffectOutput drops five write-only fields (EffectID, EffectIndex, LedgerClosed, LedgerSequence, AddressMuxed) and the per-effect fmt.Sprintf that populated them; nothing reads them. - Transaction hashes are hex-encoded at the rare log/error sites instead of eagerly per operation. - Soroban-only processors check the operation type before installing their duration-metric closure, so classic operations pay neither the timestamps nor the label lookup; invalid-op-type calls no longer land in the histogram. - ContractDeploy's dedupe map allocates on first deploy, not per call. - Zero-effect operations return before the master builder is built.
Deriving a SAC contract ID costs a SHA-256 over the asset's contract-ID preimage plus a strkey encode, and both the effects and token-transfer processors recompute it for the same few assets on every event. Each processor now memoizes by asset identity; entries are content-derived under the processor's fixed network passphrase, so they never invalidate, and sync.Map keeps the lookup safe across the indexer's pool workers.
The sponsorship effect paths had no coverage: no test exercised addSignerSponsorshipEffects, addLedgerEntrySponsorshipEffects, or the accountHasSponsoredSigner guard that spares the SponsorPerSigner build for accounts sponsoring nothing. Covers every transition (created, removed, updated, unchanged, whole-entry removal), the guard's over-admit-never-over-reject contract, deterministic signer ordering, and each sponsorable entry type's details shape.
Every NewStateChangeBuilder and Clone() heap-allocated a ~480-byte builder, and the hot paths clone constantly — a plain payment paid four heap builders for three state changes. The builder is now used by value: With* methods mutate their own copy and return it, branching is plain assignment, Clone() is gone, and builders stay on the stack. The write-only metricsService field is dropped from the builder along with the constructor parameter. parseTrustline now returns the builder it produced so the balance-authorization change emitted for a new trustline keeps inheriting the resolved token and the trustline limit exactly as it did through the shared pointer; whether that inherited limit belongs on a BALANCE_AUTHORIZATION row is flagged in the PR as a follow-up decision. Helper signatures across the effects, token-transfer, contract-deploy, SAC, SEP-41, and Blend processors take the builder by value; output is byte-identical everywhere.
The AddressMuxed removal left github.com/guregu/null unused module-wide, and the sponsorship fixtures always build entries for sponsoredAccount, so the account parameter comes off both helpers (unparam).
…tion XDR encoder The signers-only skip asked the SDK's AccountChangedExceptSigners, which deep-copies both account entries, XDR-marshals each, and compares the bytes — 2.6% of ingest CPU and 5.7% of allocated bytes at ~18k tx/s. A local field comparison preserves the exact semantics (only the Signers slice is ignored; NumSubEntries and V2 signer-sponsoring IDs still count; a missing extension equals V1 with zero liabilities), pinned by a 25-case differential test against the SDK function as oracle. The always-nil error plumbing collapses with it. ConvertOperation's per-operation XDR marshal allocated a fresh encoder and growth-doubled buffer each call; a pooled EncodingBuffer reuses the scratch across pool workers and allocates only the retained, exact-size copy of each operation's bytes.
The quadratic scan's justifying comment — streams are a handful of changes per transaction — is true for the indexer's per-transaction call but false for the sep41 and blend processors, which assign ordinals over a whole staged migration window at persist time, inside the CAS-guarded window transaction, with nothing bounding the slice but the window. A counting map keeps the assignment linear at every call site; IDs are byte-identical to the scan's. The per-call map allocation is noise on the live path: four calls per transaction against a process stage that allocates five orders of magnitude more per ledger.
…s into functions CreateContractV1OpProcessor, CreateContractV2OpProcessor, and InvokeContractOpProcessor each held one field, had one method, and were constructed and consumed on adjacent lines inside participantsForSorobanOp's switch — with no other consumer in the tree. Each method also re-checked the op type and host-function type the switch had already decided, and re-copied the op body. They are now unexported functions taking the network passphrase and the InvokeHostFunctionOp the caller already holds.
…n table missed Three accountChangedExceptSigners branches had no reaching table row in TestAccountChangedExceptSigners_MatchesSDK: a differing AccountId, an InflationDest transitioning set to nil, and sponsoring-ID slices of different length (nil vs one element). Each new row differs from the base entry in exactly one field, so its verdict can only come from that field's guard; the SDK oracle agrees on all three.
Only Soroban host functions write ContractData entries, so transactionContractDataChanges now returns before walking or allocating anything when the transaction carries no SorobanTransactionData — the common case on a payment-dominated ledger. The per-transaction wrapper lookup becomes a dense slice instead of a map, and the result map is allocated on the first change actually collected instead of eagerly. The reader-extraction equivalence test still passes over the real-ledger fixtures, which pins that the gate drops nothing a Soroban transaction produces.
…m one memo AssetContractIDMemo is the package's single cache for the SHA-256-plus- strkey derivation: token_transfer's hand-inlined sync.Map becomes a memo field, and the two SAC comparison sites that recomputed the derivation per event (isSACContract, trustlineEntryMatches) now read through a memo on the processor. Three accessors normalize the three caller shapes onto one complete key, so the same asset lands on one entry no matter which path reaches it; the key's completeness and the accessor split's cost reasoning are documented on the type. Trustline matching compares contract IDs strkey-encoded — equivalent by injectivity — which retires a per-call decode. The preimage path in contract_operations stays uncached: it fires roughly once per asset lifetime (SAC deploys), not per event.
aditya1702
force-pushed
the
live-ingest-processor-perf
branch
from
August 22, 2026 13:40
c4aa67e to
f6407a2
Compare
aditya1702
force-pushed
the
live-ingest-persist-path
branch
from
August 22, 2026 13:40
fade6bb to
2d00170
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review commit-by-commit, in order. ~1 hour. No schema changes, no new flags, no behavior changes. Every risky rewrite has a differential test against the implementation it replaced.
Why
After #684, the process stage was the bottleneck for the p99 ≤ block-time target, and GC was 20–28% of ingest CPU. This PR only cuts CPU and allocations on that stage.
What changed
State-change ordinals are assigned with a counting map — linear even for the migration path, which passes a whole window at once.
How it's verified
Plus 583 lines of new sponsorship-effects coverage (previously none).
Numbers
Combined loadtest rig result for the campaign: 18,458 tx/s, process p99 0.992 s — this PR took process p99 under the 1 s bar. Individual PRs were not re-benchmarked.
Stack
Third of four PRs from the loadtest campaign (replaces #682). Review order: #679 → #684 → this. #686 is independent.
Follow-up, decided separately: the BALANCE_AUTHORIZATION change for a new trustline inherits
trustline_limit_newfrom its sibling trustline change. Preserved byte-identically here; the field arguably doesn't belong there.