Re-land the three non-Blend commits from main-blend onto main by aditya1702 · Pull Request #730 · stellar/wallet-backend

aditya1702 · 2026-09-03T23:46:52Z

To review this: run the three range-diffs below. Check each commit matches what was already merged, minus the Blend parts.

Merge with rebase-merge, not squash — reason at the bottom.

Why this PR exists

Three PRs were reviewed and merged into main-blend and never reached main. Blend is paused, the persist stack is moving to main, and it depends on all three. None of them is a Blend feature.

Commit	Original	Size	vs the merged original
`86193aca`	#657	18 files, +1309/−182	verbatim
`f7dd79d1`	#694	2 files, +432/−33	2 dead lines dropped
`84def4fd`	#692	28 files, +1835/−330	Blend arm removed

git range-diff 5b814edf~1..5b814edf 86193aca~1..86193aca   # #657
git range-diff 08b23c2d~1..08b23c2d f7dd79d1~1..f7dd79d1   # #694
git range-diff af524d52~1..af524d52 84def4fd~1..84def4fd   # #692

#657 comes back =. #694 and #692 come back ! — their real differences are below, and the rest is context drift from main's #695 and #723.

What I dropped

#694 — 2 lines, both dead on main: an ingest/sac import main already carries, and an assertion that FetchSACMetadata is not called (#723 deleted that service). Its 8 hot-archive tests are unchanged and pass.

#692 — 4 Blend-only paths:

internal/data/blend/wipe.go
internal/data/blend/wipe_test.go
internal/integrationtests/blend_test.go
internal/services/blend/processor.go   (10-line hunk)

No call site moves: --rebuild reaches each protocol through the WipeCurrentState interface method rather than a registry, so deleting the Blend package deletes its implementation and nothing else refers to it. sep41 stays as the live implementer.

#657 is not Blend code, despite its title. It adds the shared framework — RequiresContractData() on ProtocolProcessor, ContractDataChanges on ProtocolProcessorInput, sep41 implementing it as false — and touches nothing under blend/. Every protocol on main returns false, so ContractData extraction stays off.

Nine commits are new — review these normally

Review on this PR found four bugs, one stale doc, and three duplications, fixed on top so the range-diffs still compare cleanly.

291aedc7 — an invalid contract id printed %!w(<nil>) instead of its byte count: %w wrapped a nil error.
2b527820 — DeleteNamespaceLedgerRange packs its ledger bounds into a toid, which holds the ledger in the high 32 bits of a signed int64. A bound above MaxInt32 wrapped negative and matched no rows, so the wipe reported success having deleted nothing.
5c727e6c + c40bfff1 — the migrate advisory locks were taken once and never re-checked. A CNPG failover releases them server-side while pgxpool keeps serving healthy connections, so the run writes on while a second run takes the same locks. The rebuild wipes (cursor reset, truncate, every history delete slice) now begin their transactions on the lock-holding connection: a transaction on that session cannot commit once the session, and with it the lock, is gone. Only the wipes need it: the fold loop commits through the cursor CAS, so a second run there loses its CAS and hands off. TestAcquireMigrateLocks terminates the lock backend server-side and asserts the transaction fails.
ada255f5 — the hot-archive pass skipped archived SAC instances, so finalize dropped every live SAC balance whose instance was archived at the checkpoint. The SAC extends its instance TTL by 7 days and balances by 30, so quiet assets hit this routinely, and restoring the instance emits no balance change to recover from it. Archived SAC instances now register like live ones.
84aa0215 — adding-a-protocol.md documents RequiresContractData, WipeCurrentState and ContractDataChanges.
d607a401 — generateAdvisoryLockID and migrateAdvisoryLockID hash through one advisoryLockID(key); keys unchanged.
941406b0 — both rebuild services shared one validate body; it lives on the engine and reads the status column through the strategy.
93a89987 — the SEP-41 processor wrapped its wipe error with the same text the data layer already used.
f4c66ef2 — the history rebuild takes ProtocolMigrateHistoryConfig plus the state-changes model instead of a copied config struct.

Merge method

Rebase-merge, not squash. #679 and the persist stack sit on this branch and shed their own copies of these three commits by patch-id once the commits reach main. A squashed commit matches none of the three patch-ids, so the stack would replay all three onto a tree that already contains them.

Copilot

Pull request overview

Re-lands shared non-Blend protocol infrastructure, checkpoint archival support, and protocol-state rebuilding.

Changes:

Adds ContractData extraction and live-frontier migration gating.
Loads protocol metadata from hot archives during checkpoints.
Adds locked --rebuild workflows for history and current state.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`internal/services/sep41/processor.go`	Implements new processor capabilities and wipe support.
`internal/services/protocol_processor.go`	Extends the shared processor interface.
`internal/services/protocol_migrate.go`	Adds frontier gating and ContractData processing.
`internal/services/protocol_migrate_test.go`	Tests migration gating and extraction.
`internal/services/protocol_migrate_rebuild_test.go`	Tests rebuild workflows.
`internal/services/protocol_migrate_lock.go`	Adds per-protocol advisory locks.
`internal/services/protocol_migrate_lock_test.go`	Tests lock acquisition and release.
`internal/services/protocol_migrate_history.go`	Locks history migrations.
`internal/services/protocol_migrate_history_test.go`	Tests history lock exclusion.
`internal/services/protocol_migrate_history_rebuild.go`	Implements history rebuilding.
`internal/services/protocol_migrate_current_state.go`	Locks current-state migrations.
`internal/services/protocol_migrate_current_state_test.go`	Tests current-state lock exclusion.
`internal/services/protocol_migrate_current_state_rebuild.go`	Implements current-state rebuilding.
`internal/services/mocks.go`	Updates processor mocks.
`internal/services/ingest.go`	Exposes materialized transactions.
`internal/services/ingest_live.go`	Integrates ContractData into live processing.
`internal/services/ingest_live_test.go`	Updates ingest configuration coverage.
`internal/services/ingest_backfill.go`	Uses the canonical oldest cursor.
`internal/services/checkpoint.go`	Processes hot-archive entries.
`internal/services/checkpoint_test.go`	Tests archived entry handling.
`internal/metrics/ingestion.go`	Corrects protocol-state metric help.
`internal/integrationtests/data_migration_test.go`	Updates protocol-contract lookup.
`internal/ingest/timescaledb_test.go`	Uses the canonical cursor name.
`internal/ingest/ingest.go`	Removes configurable oldest cursor plumbing.
`internal/ingest/datastore_backend.go`	Updates obsolete documentation.
`internal/indexer/indexer.go`	Adds ContractData extraction helpers.
`internal/indexer/indexer_test.go`	Validates extraction against fixtures.
`internal/data/statechanges.go`	Adds namespace-range history deletion.
`internal/data/statechanges_test.go`	Tests scoped history deletion.
`internal/data/sep41/wipe.go`	Adds SEP-41 current-state truncation.
`internal/data/sep41/wipe_test.go`	Tests SEP-41 wipe boundaries.
`internal/data/protocol_contracts.go`	Supports transaction-scoped reads.
`internal/data/mocks.go`	Updates protocol-contract mocks.
`internal/data/ingest_store_test.go`	Updates canonical cursor fixtures.
`docs/data-migrations/running-a-data-migration.md`	Documents rebuild operations.
`cmd/protocol_migrate.go`	Adds `--rebuild` commands.
`cmd/ingest.go`	Removes the obsolete cursor flag.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Suppressed comments (2)

internal/services/protocol_migrate_lock.go:49

The advisory lock is session-scoped, but this connection is not checked again until release. If a database failover drops the session, PostgreSQL releases these locks while the migration continues through separate pool connections; another migration/rebuild can then acquire the same key and run concurrently, including wiping state under the first run. Keep the lock session monitored for the whole operation (as live ingestion does with checkLockSession) and cancel/abort the migration immediately if it dies.
internal/services/protocol_migrate.go:719
When decoding succeeds but the length is wrong, err is nil, so wrapping it with %w produces a malformed %!w(<nil>) suffix. Handle decode errors and length validation separately so malformed stored IDs report a clear byte-length error.

			if err != nil || len(idBytes) != len(xdr.ContractId{}) {
				return nil, fmt.Errorf("protocol %s contract id %q is not a 32-byte hex hash: %w", t.protocolID, c.ContractID, err)

Copilot

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated no new comments.

…or Blend processors (#657) * feat(protocols): add RequiresContractData capability gate to ProtocolProcessor * feat(indexer): add ExtractContractDataChangesForLedger * feat(protocols): extract ContractData changes in the migration engine behind the capability gate * feat(ingest): thread ContractDataChanges through live ingestion behind the capability gate * fix(ingest): resolve full protocol membership for ContractData processors Live ingestion derived ProtocolContracts from this ledger's event emitters only. A ContractData-requiring processor needs the protocol's complete committed membership: entries can change on contracts that emitted no event this ledger, and event decoding disambiguates shared symbols against the full tracked set (e.g. backstop vs pool withdraw). * fix(protocols): fail fast on contract id encoding and split contract-data extract metric phase * fix(ingest): persist protocol wasm rows before protocol processors run A processor enriching protocol_contracts (contract names decoded from instance storage) inserts rows FK-filtered against protocol_wasms; a contract deployed in the same ledger as its wasm upload was silently dropped because the wasm rows persisted after the processor block. * fix(protocols): refresh contract membership per window for ContractData processors The migration engine loaded each tracker's classified-contract membership once before the unbounded ledger loop. A contract classified while the run is in flight (live ingestion's validator classifies newly deployed contracts concurrently) never reached membership-driven processors: for BLEND, a pool deployed mid-run lost its history rows permanently and mis-resolved the pool-vs-backstop withdraw disambiguation for the rest of the run. Trackers whose processor requires ContractData now re-read membership via GetByProtocolID after every committed window (both in-loop and at-tip flushes), mirroring the live path's per-ledger full-membership resolution. Event-only processors keep the run-start snapshot. * perf(ingest): reuse the staging pass's transactions for ContractData extraction Live ingestion materialized every ledger's transactions twice: once in the main indexing pass and again inside ExtractContractDataChangesForLedger, whose reader constructor re-hashes every transaction envelope. ProcessLedger now returns the transactions it already built, and the live path hands them to a new ExtractContractDataChangesFromTransactions, so a ledger is read exactly once. The ledger-based extractor remains as a thin wrapper for protocol-migrate, which has no prior transaction pass to share. * perf(protocols): footprint-gate ContractData extraction, drop the reader protocol-migrate spent ~7ms/ledger in extract_contract_data — a LedgerTransactionReader build (SHA-256 of every envelope) plus a full GetChanges walk of every transaction — on every ledger of the range, though almost none touch a tracked contract. Soroban guarantees writes ⊆ the declared read-write footprint (host storage is footprint-seeded and writes outside it trap; RestoreFootprint restores exactly the read-write keys; protocol-23 auto-restores are indices into it), so a skim of the already-decoded envelopes' footprints decides ledger relevance exactly. Ledgers whose footprints touch no tracked contract skip extraction outright; the rare hit extracts via transaction meta directly — GetChanges reads only meta/result/ledger-version, so no reader is built even then. The fixture-corpus test now pins both properties on real ledgers: meta-based output equals the reader-based reference, and every changed contract, tracked alone, triggers the gate. Live ingestion keeps the ungated slice-based path over its already-materialized transactions. * fix(indexer): populate Hash on synthesized LedgerTransactions GetChanges attaches the transaction to every Change it returns, so the footprint-gated extraction path handed processors transactions with a zero Hash while the reader-fed path populates it. Mirror the reader: resultPair.TransactionHash is the exact value it reads. Also scope the equivalence-test comment to the change fields the fixture actually compares — the attached transaction is projected away. * test(services): drop mock-only RequiresContractData test It asserted a testify mock returns what it was programmed to return; the gate is covered behaviorally by protocol_migrate_test.go (populated vs left nil) and ingest_test.go cases H/I/J, and the compile-time interface assertion already exists in mocks.go. * fix(protocols): read window membership after the window's first ledger is fetched The engine refreshed a tracker's classified-contract membership right after each window commit — milliseconds before a concurrent live transaction for that same contested ledger finishes committing a new contract's classification, since live's lost CAS was blocked behind the engine's row lock and still has work to do after unblocking. The engine then staged the next window with a snapshot missing that contract, skipping its events and entries for one more ledger; additive fold columns (cost basis, lifetime claimed totals) never heal from a missed ledger, so those deltas were permanently lost whenever the engine also won that next ledger. The refresh now runs at window start, after GetLedger returns for the window's first ledger: the fetch blocks until that ledger has closed, which is a full ledger interval after any concurrent transaction for the previous one committed. Same cadence — once per window per requiring tracker — with the read taken at the latest useful moment. The refresh test now pins the ordering: the first folded ledger must already carry membership committed after the run-start snapshot (mutation-verified — removing the window-start refresh fails it). * feat(migrate): gate the fold loop on live ingestion's classification frontier The engine could fold ledgers live ingestion had not yet reached. Live is the sole classifier, so a contract deployed in that lead was invisible to the engine's membership for the rest of its lag window, and cursor-passed ledgers are never replayed — protocol state for that span was permanently lost (#680). Live's cursor advances in the same transaction that commits a ledger's protocol_contracts classifications, so cursor >= seq is atomic proof that every classification discoverable at seq is committed. The fold loop now waits on that frontier before fetching seq: one cached comparison during the bulk backfill, a logged 1s poll at the frontier. The gate flushes open windows before waiting, so no window outlives the cursor value that admitted it — every folded ledger sees complete membership — and the committed cursor reaches the last folded ledger, arming a clean CAS handoff the moment live arrives. The gate subsumes flushWindowsAtTip: the live cursor never exceeds the chain tip, so the frontier flush always fires first, and the engine's only GetLatestLedgerSequence call disappears with it. This also makes a CAS winner's membership complete by construction — a ledger live loses is folded later by an engine that sees everything live's transaction committed, including contracts classified that very ledger — which is what allowed removing repairClassificationGap and its one-ledger-scoped healing of exactly this gap. Fixes #680 * chore(metrics): drop stale load_current_state phase from Help text The protocol state processing histogram's Help enumerated a load_current_state phase that no code path observes; the emitted phases are process_ledger, persist_history, and persist_current_state. * fix(migrate): pin every window to the frontier value that opened it The gate's fresh-read path could return without flushing: a window opened under cursor value C could keep folding ledgers past C as live advanced, on the membership snapshot taken at the window's start. A contract classified in that stretch was invisible to the snapshot — its footprint skipped, its events filtered — and the window's CAS then passed those ledgers forever: the #680 loss shape, inside a window. The gate now closes every open window whenever seq crosses the cached frontier, before it even reads the new value, so no window ever contains a ledger past the cursor value current when it opened — which is exactly the bound the window-start membership refresh covers. flushWindow no-ops when nothing is pending, so bulk-backfill crossings (once per cache refill) cost one committed window and frontier crossings are the intended shrink-to-one cadence. Membership refresh now also runs for event-only trackers, on the same window-start cadence: their processors filter events by membership too, so the run-start-only snapshot had the same silent-drop failure through a different door. The regression test replays the exact scenario: live classifies a contract mid-window; the crossing must flush the open window and the next window must fold with the refreshed membership. Mutation-verified against the fresh-read early return.

…e stored (#694) PopulateFromCheckpoint reads only the live bucket list, so contract code and contract instances evicted under protocol-23 state archival never reached protocol_wasms/protocol_contracts. Archived SEP-41 contracts therefore stayed unclassified and their current state was never computed, producing wrong balances once those contracts were restored. Add a hot-archive pass (ingest.NewHotArchiveIterator) inside the same load transaction, before finalize: archived ContractCode joins wasmClassifications and archived wasm-executable contract instances land in contract_tokens and protocol_contracts. Archived balance entries and SAC instances are skipped; live ingestion recreates their rows on restore. Checkpoints predating protocol 23 (HAS version < 2) skip the pass instead of failing on the absent hot-archive bucket list. Classification needs no changes: stellar-rpc (protocol 23+) getLedgerEntries returns archived entries with bytecode, so the RPC-based setup pass resolves archived wasm hashes as-is.

…ate (#692) * feat(migrate): exclude concurrent current-state writers with a per-protocol advisory lock Current-state migrations (and the upcoming rebuild mode) must not write the same protocol's tables concurrently. Each run try-locks a per-protocol advisory lock on a dedicated connection for its duration; a held lock fails the run before any status is marked. Live ingestion is deliberately not a party — the per-ledger cursor CAS already arbitrates which writer folds a given ledger. Plain history migration is append-only and does not contend. * feat(migrate): rebuild modes — current-state wipe+remigrate and history bounded backfill Current-state rebuild (engine.rebuild): after taking the per-protocol advisory lock, one transaction per protocol resets the migration cursor to start-ledger − 1 and deletes every current-state row via the processor's WipeCurrentState. The cursor UPDATE takes the row lock live ingestion's per-ledger CAS needs, so live serializes against the wipe and skips the protocol's folds until the migration hands ownership back at the frontier. validate() re-admits protocols whose migration already succeeded and refuses in_progress residue. History rebuild (bounded fold): history rows are per-ledger records with no running totals, so any range rebuilds in isolation. The service validates a COMPLETED history migration, clamps the range to [oldest retained ledger, committed history frontier], deletes the protocol's state_change_id-namespace rows in 10k-ledger slices (each its own transaction with the DML decompression cap lifted; to_id carries the ledger in its high 32 bits, so chunk skipping and compressed-batch minmax metadata prune every slice), then re-derives the range through the migrate pipeline with no cursor reads/writes, no live-frontier gating, and no CAS. Wipes never touch contract_tokens, protocol_wasms, or protocol_contracts — classification owns those and nothing rebuilds them. * feat(migrate): expose the rebuild modes as --rebuild flags current-state gains --rebuild; history gains --rebuild with optional --from-ledger/--to-ledger (valid only with --rebuild, defaulting to the full retained window). Help text states the destructiveness plainly. The history rebuild honors --oldest-ledger-cursor-name the same way the plain history migration does. * fix(migrate): cover the final ledger's last transaction in the history delete toid.AfterLedger is the maximum to_id within a ledger, not the first of the next one, so the upper bound must be inclusive. Also wrap the errors the wipe wrappers return. * test(migrate): pin the rebuild semantics end to end Unit coverage: current-state rebuild re-admits succeeded protocols, wipes exactly once, and resets the cursor in the same transaction (a failed wipe rolls the reset back); in_progress residue refuses before any wipe; non-rebuild behavior unchanged. History rebuild: status/classification validation, range clamping to [oldest retained, lowest committed frontier], 10k-ledger delete slicing verified at every slice edge, bounded folds that never touch the cursor, and advisory-lock refusal. Data layer: namespace ∩ ledger-range deletes across all three namespaces, idempotent on retry. Integration: BlendMigrationTestSuite gains a phase that deletes one migrated BORROW state change, runs protocol-migrate history --rebuild over exactly that ledger, and asserts the row returns with other namespaces, contract_tokens, the cursor, and both migration statuses untouched. * refactor(ingest): hard-code the oldest-ledger cursor name The cursor name was configurable via --oldest-ledger-cursor-name on both ingest and protocol-migrate history, but nothing ever needed a non-default value, and a mismatch between the two commands silently breaks history migration's start-ledger resolution. The key is now the data.OldestLedgerCursorName constant everywhere, matching the hard-coded latest-ledger cursor. * test(migrate): cover the corrupt-value shape in the history rebuild scenario The rebuild exists for rows that are present with wrong values, not just missing ones. The integration phase now corrupts the REPAY amount in place, deletes the BORROW row, rebuilds the multi-ledger range spanning both, and asserts the corrupted value is replaced by the re-derived one and the deleted row returns. Because state_change_ids are deterministic, the corrupted row's repair specifically witnesses the rebuild's delete step — a silently failing delete would leave the wrong amount in place. * test(migrate): pin the advisory-lock semantics directly The lock had only indirect coverage through the migration and rebuild Run paths. Pin the primitive itself: deterministic per-protocol IDs that differ across protocols, refuse-while-held with release restoring acquirability, no contention between distinct protocols, and — the cleanup path — a multi-protocol acquisition failing mid-list frees the locks it already took instead of wedging every protocol before the held one. * refactor(migrate): per-strategy locks and symmetric rebuild services Every command's Run now takes its own strategy's per-protocol advisory lock (current-state scope or history scope) and the engine takes none — it is pure migrate machinery again, with no lock flag, no rebuild flag, and no wipe logic. History migrations gain mutual exclusion they previously lacked. The current-state rebuild moves out of the engine into its own service, mirroring the history rebuild: validate → lock → wipe → re-derive. Its wipe transaction resets the migration status to not_started alongside the cursor and the row deletes, which is what lets it reuse the engine's normal lifecycle unmodified. The current-state lock key bytes are unchanged; the history scope adds a new key. * refactor(migrate): give the history rebuild its own replay loop The bounded fold shared only the shape of the engine loop, none of its machinery — cursor init, frontier gating, CAS, handoff, and per-window membership refresh are all live-race concerns a below-frontier replay never has. reDerive is a plain fetch/extract/process/persist loop over the inclusive range, loading membership once (classification for every ledger at or below the frontier committed before the run started). The engine loses its bounded mode and returns to exactly its pre-rebuild form. * refactor(migrate): make the history rebuild a full-window remigration, mirroring current-state The history rebuild now shares the current-state rebuild's shape exactly: validate → lock → wipe → run the unmodified migration engine. Its wipe resets the protocol's cursor to the retention floor and the migration status to not_started (one transaction, committed before any delete so live ingestion stops writing the protocol's history and the deletes race nothing), then removes the protocol's rows over the retained window in ledger slices. The engine folds from the floor and hands off to live at the frontier, exactly as a first migration does. This drops --from-ledger/--to-ledger and the dedicated replay loop. Range scoping traded per-fix speed for symmetry; it returns if history migration is later folded into backfilling, where the range-scoped delete+re-derive is the natural shape. * docs(migrate): explain why the history wipe deletes in ledger slices * perf(migrate): truncate current-state tables instead of deleting every row Live ingestion writes every protocol in one transaction per ledger, and that transaction CASes each protocol's ingest_store cursor row. The current-state rebuild's wipe holds that same row while it clears the protocol's tables, so the wipe's duration is a stall for all ingestion, not just the protocol being rebuilt — and a full-table DELETE's duration grows with the row count. TRUNCATE's cost does not, which bounds the stall. One statement per protocol also takes all the table locks at once rather than escalating through them. Safe for these tables: nothing foreign-keys them (sep41_balances and sep41_allowances reference contract_tokens, not the reverse), and none are hypertables. * test(migrate): cover the current-state wipes against real tables Both wipes were only ever exercised through a recording processor that incremented a counter, so omitting a current-state table — or clearing a classification table — would have gone unnoticed. The Blend test asserts coverage by filenode rather than by seeding all eleven tables: TRUNCATE rewrites a table's underlying file, so a changed filenode proves the wipe reached it and an unchanged one proves it did not. That also catches a current-state table added later and forgotten, which seeded rows — listing only the tables the test itself knows about — would not. * fix(migrate): read the history wipe's upper bound after the cursor reset The bound came from a single latest_ingest_ledger read taken before any protocol's cursor was reset. Live could commit history above it and those rows survived the delete, leaving the re-migration to re-derive them at the same deterministic state_change_ids. On a multi-protocol run the gap spanned every preceding protocol's entire delete loop. Reading it per protocol, after that protocol's reset commits, closes the window: a live writer that already passed its CAS holds the cursor row, so the reset waited for it to commit, and live bumps latest_ingest_ledger in the same transaction it writes history. Writers after the reset fail their CAS. Also refuse an inverted window. The slice loop advances by uint32 arithmetic that never reaches an upper bound below its start, so an operator mistake ran it until the process was killed. * test(migrate): pin the advisory lock IDs to golden values The determinism assertion compared the function with itself, so it stayed green through any change to the prefix, the hash, or the cast — exactly the changes that would let a rolling deployment run old and new binaries taking different locks for the same protocol. * docs(migrate): document rebuilds and drop the removed cursor-name flag The operator guide still advertised --oldest-ledger-cursor-name, so following it failed with an unknown flag. It also said nothing about --rebuild: what each wipe clears, which tables are never touched and why, the locking, and why the two rebuilds order the wipe differently against the cursor reset. * fix(migrate): cap how long the current-state wipe waits for a lock TRUNCATE needs ACCESS EXCLUSIVE, so a long-running API read on one of the protocol's tables makes it wait — and Postgres queues later lock requests behind the waiting TRUNCATE, so readers pile up too. All of that happens while the transaction holds the ingest cursor row, which stalls live ingestion for every protocol. SET LOCAL lock_timeout = '5s' turns that open-ended stall into a failed run the operator can retry. * fix(migrate): refuse an unusable history window before resetting anything The inverted-window check ran inside wipe, after that protocol's cursor reset had already committed. Failing there left the protocol with its cursor at the retention floor and its status not_started, which stops live from writing its history until a migration runs — a worse state than the one the operator started with. The check moves up to oldestRetained, before any protocol is touched. Each wipe still reads its own upper bound after its reset, since only then is it authoritative. The slice loop now bounds itself on start <= latest instead of relying on that check, so it cannot spin on uint32 arithmetic regardless of what it is handed. * docs(migrate): clarify that a wipe covers only the named protocol The table read as though a current-state rebuild clears the sep41 and blend tables together, when each run clears only the tables of the protocol passed to --protocol-id.

…ror wrap A contract id that hex-decodes cleanly but is not 32 bytes left err nil, so the %w verb rendered as %!w(<nil>) — the operator saw a format artifact instead of the actual length. Decode failures and wrong lengths are now separate messages, and the length one reports the byte count it got.

toid packs the ledger into the high 32 bits of a signed int64, so a bound above MaxInt32 wrapped negative and produced a to_id range matching no rows: the wipe would report success having deleted nothing, and the re-migration would then collide on state_change_ids already present. The bounds reach this function from ingest_store's parsed cursor values, which is why CodeQL flags these two conversions and not the XDR-sourced ones elsewhere.

The migrate locks are session-scoped and were taken once, then never probed. A CNPG failover ends that session server-side, releasing every lock, without this process seeing the disconnect — pgxpool keeps handing out other, healthy connections, so the run keeps writing while a second run acquires the same keys. Live ingestion already probes its lock session for exactly this reason. acquireMigrateLocks now returns the held set with a checkSession probe alongside release, and the two rebuild wipes call it: current-state before its truncate transaction, history before the cursor reset and again before every delete slice. Only the wipes. The fold that follows commits each window through the cursor CAS, so a second run there loses its CAS and hands off — the fold path is already safe against a lost lock. The wipes have no CAS behind them, and a run deleting rows another has already re-derived loses them for good, since the deleting run's cursor reports the range done.

…es survive the load The hot-archive pass skipped archived SAC instances, so finalize's verified-SAC filter dropped every live balance entry whose SAC instance was archived at the checkpoint. The SAC extends its instance TTL by 7 days and balances by 30, so a quiet asset's instance archives while holder balances stay live, and restoring the instance emits no balance change that live ingestion could fold. Register the archived instance like a live one; it has no wasm, so no protocol_contracts mapping is written.

…ntractDataChanges The adding-a-protocol guide listed the ProtocolProcessor interface without the two methods --rebuild and ContractData-driven processors rely on, or the input field that carries the ContractData changes. Add them to the interface listing, the responsibilities table, the example processor, and the key patterns.

Probing the lock session and then writing through another pooled connection left a window: a CNPG failover between the probe and the write ends the session, releases the lock, and lets a second run take it while the first still truncates or deletes. A transaction begun on the lock-holding connection cannot commit once that session is gone, so the cursor reset, the truncate, and every history delete slice now begin there. The session probe goes away with the window it guarded.

generateAdvisoryLockID and migrateAdvisoryLockID each ran their own FNV-64a over a wire-level key string. One helper takes the key so the two cannot drift; the keys themselves are unchanged.

Both rebuild services carried the same validate body, differing only in which migration status column they read. The engine's strategy already selects that column (MigrationStatusField) and names the strategy (Label), so one engine method serves both.

sep41data.WipeCurrentState already wraps with "wiping SEP-41 current state", and the rebuild adds its own prefix, so the processor's wrap repeated the same fact.

…onfig ProtocolHistoryRebuildConfig repeated every field of ProtocolMigrateHistoryConfig and the constructor copied them across one by one. The rebuild now takes the migration config itself plus the state-changes model its wipe needs, the same shape as the current-state rebuild.

Copilot

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 1 comment.

+	if latest < oldest {
+		return 0, fmt.Errorf("latest ingest ledger %d is below the oldest retained ledger %d: refusing to rebuild an inverted window", latest, oldest)
+	}
+	return oldest, nil


Copilot AI balanced review requested due to automatic review settings September 3, 2026 23:46

Copilot started reviewing on behalf of aditya1702 September 3, 2026 23:47 View session

github-advanced-security Bot found potential problems Sep 3, 2026

View reviewed changes

Comment thread internal/data/statechanges.go Fixed

Comment thread internal/data/statechanges.go Fixed

Copilot AI reviewed Sep 3, 2026

View reviewed changes

Comment thread internal/services/protocol_migrate_lock.go Outdated

Comment thread internal/services/protocol_migrate.go Outdated

aditya1702 force-pushed the reland/migrate-rebuild-and-hot-archive branch from ea6f71d to 6e5545f Compare September 4, 2026 19:30

Copilot AI review requested due to automatic review settings September 4, 2026 19:30

Copilot started reviewing on behalf of aditya1702 September 4, 2026 19:31 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 8, 2026 18:53

Copilot started reviewing on behalf of aditya1702 September 8, 2026 18:53 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

aditya1702 force-pushed the reland/migrate-rebuild-and-hot-archive branch from 25f92cd to 3835f9c Compare September 8, 2026 19:03

aditya1702 requested a review from aristidesstaffieri September 8, 2026 19:13

aristidesstaffieri reviewed Sep 8, 2026

View reviewed changes

Comment thread internal/services/checkpoint.go Outdated

aristidesstaffieri reviewed Sep 8, 2026

View reviewed changes

Comment thread internal/services/protocol_processor.go

aditya1702 force-pushed the reland/migrate-rebuild-and-hot-archive branch from 3835f9c to 3a8b433 Compare September 9, 2026 21:03

Copilot AI review requested due to automatic review settings September 9, 2026 21:03

Copilot started reviewing on behalf of aditya1702 September 9, 2026 21:03 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Comment thread internal/services/protocol_migrate_lock.go Outdated