Skip to content

Fail indexer batch on unresolved deactivation reference - #599

Draft
scolear wants to merge 1 commit into
digital-asset:mainfrom
scolear:fix/indexer-null-deactivation-backpointer
Draft

Fail indexer batch on unresolved deactivation reference#599
scolear wants to merge 1 commit into
digital-asset:mainfrom
scolear:fix/indexer-null-deactivation-backpointer

Conversation

@scolear

@scolear scolear commented Aug 3, 2026

Copy link
Copy Markdown

Fixes #598.

What

The indexer computes the deactivated_event_sequential_id back-pointer for each deactivation event at ingestion time. When resolution failed, it logged a warning and persisted the event with a NULL back-pointer. Every Ledger API activeness query anti-joins on that back-pointer, so the archived contract stayed visible as active in the ACS permanently. We observed this in production: a mainnet participant served 3 archived contracts as active for 7 weeks, diverging from every other participant hosting the same party (details in #598 and DA support ticket DA-4793).

This PR makes the failure loud instead of corrupting:

  1. dbPrepare now fails the ingestion batch when any deactivation's activation reference stays unresolved, following the ReferencedContractNotFoundException recovery pattern: the indexer restarts and retries. Before throwing, it invalidates the cached contract lookups for the affected contracts, so a stale cached mapping cannot cause a restart loop.
  2. The tolerant branch in refillMissingDeactivatedActivations becomes an invariant violation, closing the only code path that wrote NULL back-pointers.
  3. A new integrity check reports deactivation events with a NULL activation reference at or above the pruning point and below the ledger end. The existing checks either skipped NULL back-pointers or treated them as pruning leftovers.
  4. A new Errors-qualified counter, indexer_restart_due_to_unresolved_deactivation, makes the restarts alertable.

Design decisions

Cache invalidation uses a new ContractStore.invalidateCachedContracts method instead of the existing contractsPruned. The existing method deletes contract data on the in-memory store (correct at its call site, where the contracts are proven absent), which would destroy live contracts here. The new method only invalidates caches: the JDBC store invalidates its lookup cache, the in-memory store is a no-op because it has no cache.

A persistent resolution failure now stalls the indexer deliberately. The retry converges when the failure was transient (a stale cached mapping, or a ledger-end-cache window). If the activation reference is genuinely unresolvable, the indexer keeps restarting: we consider a loud stall preferable to silently persisting corruption that pruning later makes undiagnosable. If you would rather have an operator escape hatch (bounded retries, or a flag restoring the old warn-and-persist behavior), we are happy to add one — we did not want to grow the config surface unilaterally.

Can a legitimate flow produce an unresolvable deactivation? We believe not. Pruning only removes an activation row together with the deactivation that consumed it, so a live contract's activation row is never pruned; witnessed-only consuming exercises take the lapi_events_various_witnessed path and need no back-pointer. An unresolved reference at ingestion time is therefore either transient or evidence of an inconsistent index database. The integration test environments run verifyIntegrity at teardown, so a green CI run across the ACS-import, party-replication, and pruning suites is empirical evidence for this claim — we would appreciate maintainers double-checking the reasoning.

How tested

  • community-common / ledger-api-core (main and test) pass scalafmtCheck and compile.
  • ParallelIndexerSubscriptionSpec extended: the throw from dbPrepare (both failure stages: internal-contract-id resolution miss and lastActivations miss), the invalidation contents, the single summarizing error log, the metric increment (via InMemoryMetricsFactory), the no-invalidation happy path, and the invariant violation through batcher.
  • StorageBackendTestsIntegrity extended: the new check reports a NULL back-pointer below the ledger end and at exactly the pruning point, and ignores rows beyond the ledger end. One pre-existing test comment claimed NULL back-pointers are "not reported"; that test only stayed green because the stray-deactivation check fires first, and the comment now says so.
  • Full local run: 304 tests, 0 failures across ParallelIndexerSubscriptionSpec and the complete StorageBackendSpecH2 suite.

Not covered locally, deferred to CI: the Postgres storage-backend variant, downstream module compilation, and the integration suites. Two known coverage gaps we chose not to close here because they need the full e2e rig: the production wiring of the invalidation callback is verified by compilation only, and no test pins that the new exception restarts rather than halts the indexer (the sibling ReferencedContractNotFoundException has the same gap).

User-facing changes

New metric, new integrity check message, and a behavior change (indexer restart instead of silent corruption) — documented in UNRELEASED.md under Bugfixes.

A deactivation event whose activation reference cannot be resolved was
persisted with deactivated_event_sequential_id = NULL. Every activeness
query anti-joins on that reference, so such a deactivation is invisible
and the deactivated contract stays in the Ledger API ACS permanently.
Only a warning was logged, and pruning later deletes the orphaned
deactivation row while the phantom activation survives, hiding the
corruption. Observed in production: a participant served 3 archived
contracts as active for 7 weeks, diverging from every other node
hosting the same party.

Fail the ingestion batch instead and let the indexer restart, following
the ReferencedContractNotFoundException recovery pattern. Before the
restart, invalidate the cached contract lookups for the contracts whose
internal contract id resolved, via a new cache-invalidation-only
ContractStore method (contractsPruned would delete live contracts from
the in-memory store), so a stale cached mapping cannot cause a restart
loop. A persistent resolution failure keeps the indexer restarting: a
loud stall is preferable to persisting the corruption.

Add an Errors-qualified counter metric for these restarts, and an
integrity check that reports deactivation events with a NULL activation
reference at or above the pruning point and below the ledger end.
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

✅ All required contributors have signed the CLA for this PR. Thank you!
Posted by the CLA Assistant Lite bot.

@scolear

scolear commented Aug 4, 2026

Copy link
Copy Markdown
Author

I have hereby read the Digital Asset CLA and agree to its terms

@scolear

scolear commented Aug 4, 2026

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Indexer writes deactivation events with NULL activation back-pointer on resolution failure, permanently corrupting the Ledger API ACS

1 participant