Skip to content

fix(l1): gate trie-layer commit/prune on canonical depth#6930

Open
edg-l wants to merge 4 commits into
mainfrom
fix/layer-commit-gate-on-finalized
Open

fix(l1): gate trie-layer commit/prune on canonical depth#6930
edg-l wants to merge 4 commits into
mainfrom
fix/layer-commit-gate-on-finalized

Conversation

@edg-l

@edg-l edg-l commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Problem

Full-sync ethrex nodes can wedge permanently, looping:

Full sync cannot resume: post-state for block 0 is absent (pruned from the layered store, or never executed). ... resume_parent_number=0 local_head=0

Observed on glamsterdam-devnet-6 (prysm-paired ethrex nodes). This is the Q2 half of the incident cluster — see #6921 (trigger: peers not serving headers by hash) and #6924 (Q1: syncer lock held while waiting for a forkchoice head). Q1 (the lock) was masking Q2 (this state-resume wedge); once the syncer isn't pinned, the node actively cycles and hits this.

Root cause: trie diff-layers were committed/pruned by a fixed-depth count (DB_COMMIT_THRESHOLD) regardless of canonicality. Under a stale consensus forkchoice, non-canonical newPayload blocks executed ahead of canonicalization tripped that threshold, flushing non-canonical state to the single path-keyed on-disk root and overwriting genesis. Since those blocks never canonicalized (local_head stayed 0), the full-sync resume gate (is_resume_point = is_canonical_sync && has_state_root) found no canonical+stateful block and paused every cycle.

Fix

Gate trie diff-layer disk commits on CANONICAL + DEPTH rather than depth alone:

  • In forkchoice_update (after canonicalization), the Store computes safe_commit_root = the state_root of the canonical block at head - commit_threshold, and writes it into a dedicated Arc<RwLock<H256>> cell shared with TrieLayerCache.
  • The trie-update worker's get_commitable(parent_state_root) returns that root only if it is found on the executed parent walk; otherwise it commits nothing. H256::zero() (chain shorter than the threshold, e.g. genesis init) → no commit, so genesis-on-disk is never gated away.
  • The cell is written only by the FCU path (post-canonicalization) and is disjoint from the worker's RCU cache writes — no lost-update race, no std::RwLock guard held across .await.

Consequences:

  • Non-canonical state is never committed → genesis (and the canonical floor) is preserved → the resume gate always has a canonical+stateful block. Wedge fixed.
  • Memory is bounded to ~commit_threshold layers in all cases, including indefinite non-finality (no dependence on finalization — distinct from a finalized-gated approach, which would grow unbounded on devnets / finality incidents).
  • A compile-time const _: () assert ties DB_COMMIT_THRESHOLD to REORG_DEPTH_LIMIT so a committed block can never be reorged out.

Removes the now-unused BATCH_COMMIT_THRESHOLD / is_batch / batch_mode depth-only commit path.

Testing

  • White-box gate unit tests in layering.rs, including a red→green regression (noncanonical_depth_old_gate_commits_new_gate_does_not) contrasting the old depth-only gate (commits non-canonical state) against the new gate (commits nothing while no canonical safe-commit point exists), plus a memory-bound assertion.
  • Public-API integration test test/tests/blockchain/canonical_commit_gate_tests.rs: non-canonical imports (no FCU) never prune genesis.
  • is_resume_point tests relocated to test/tests/p2p/.
  • Validated locally with the CI commands: cargo test --workspace --exclude 'ethrex-l2*' --exclude ethrex-prover --exclude ethrex-guest-program (green), make -C tooling/ef_tests/blockchain test (green — 0 "state root missing for block 0"), make -C tooling/ef_tests/engine test (green), cargo clippy --workspace … -D warnings (clean).

Relationship to #6924 / #6921

Complementary, different files (this PR: storage/blockchain; #6924: sync_manager.rs). #6924's description left this exact "Full sync cannot resume … removedb" case as a separate change, suggesting an automatic reset-and-resync; this PR takes the preventive angle (never strand the resume gate in the first place), which should make a recovery mechanism unnecessary for this cause. Worth aligning on the Q2 direction.

@edg-l edg-l requested a review from a team as a code owner June 29, 2026 10:04
@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

Findings:

  1. High: this removes the only persistence path for sync imports that do not carry a finalized head. TrieLayerCache::get_commitable() now hard-returns None while finalized_root is zero (crates/storage/layering.rs:191-196), and apply_trie_updates() now always uses that gate (crates/storage/store.rs:3311-3314). But full sync still canonicalizes batches with forkchoice_update(..., None, None) (crates/networking/p2p/sync/full.rs:689-696), and snap sync does the same (crates/networking/p2p/sync/snap_sync.rs:649-656). With BATCH_COMMIT_THRESHOLD removed, those paths no longer flush trie layers at all until some later FCU happens to include finalized. That is a real regression: memory grows with the entire unfinalized sync window, and a restart leaves a canonical/stateless suffix that must be re-executed from the last persisted state.

  2. Medium: the finalized gate is not restored from disk on startup. Store::from_backend() always seeds finalized_root with H256::zero() (crates/storage/store.rs:1731-1740), even though the DB already stores FinalizedBlockNumber (crates/storage/store.rs:1091-1094). After any restart, commits are disabled again until a fresh forkchoice_update(..., Some(finalized)) arrives (crates/storage/store.rs:2603-2605). That makes persistence depend on future CL traffic instead of the already-persisted finalized state, which can again cause unnecessary RAM growth and loss of durable progress.

Open question:

  • If the intent is “never prune unfinalized state,” I think the code still needs a separate “commit but retain newer layers” trigger for sync mode. Right now the change conflates pruning policy with whether we write anything to disk.

I couldn’t run the targeted Rust tests here because cargo/rustup tried to write under a read-only home directory.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

This PR replaces the threshold-based trie commit strategy with a finalized-root gate, ensuring non-finalized state is never persisted to disk (protecting against reorg data loss). The implementation is correct and well-tested.

Code Quality & Correctness

  1. Commit Gating Logic (crates/storage/layering.rs:173-210)
    The get_commitable function correctly implements the ancestor-walk algorithm:

    • Returns None immediately if the finalized cell is zero (CL hasn't reported finality)
    • Returns Some(finalized) if the parent is the finalized block (fast path)
    • Walks parent pointers with cycle detection (next == parent_state_root guard) and step limits
    • Returns None if the walk exhausts without finding the finalized root
  2. Layer Retention (crates/storage/layering.rs:334-361)
    The commit function correctly retains layers newer than the finalized root (retain(|_, l| l.id > top_layer_id)), ensuring speculative state above the finalized head remains in memory for potential reorgs. This is the correct behavior for Ethereum's fork choice rules.

  3. Lock Sharing (crates/storage/store.rs:231-240)
    The finalized_root Arc is properly shared between Store and TrieLayerCache via new_with_finalized, allowing forkchoice_update to update the cell without cache reconstruction.

  4. Zero Sentinel Handling (crates/storage/layering.rs:193-195)
    The explicit check if finalized.is_zero() { return None; } ensures genesis state is never pruned by the layer cache (genesis is written directly to disk and should not be subject to in-memory pruning).

Security & Consensus

  • No Consensus Impact: This change affects only the persistence layer (when to flush to RocksDB), not state computation or block validation. The EVM execution produces the same state roots; only the durability timing changes.
  • Reorg Safety: By gating on CL finality, the PR ensures that state subject to reorganization remains in memory where it can be discarded, while finalized state is persisted. This is architecturally correct for an Ethereum client.
  • Poison Handling: Lock poisoning is handled gracefully—get_commitable returns None (fails safe to no commit), and set_finalized_state_root returns StoreError::LockError.

Testing

The test coverage is comprehensive:

  • Unit tests (layering.rs): Verify cycle detection, ancestor walking, and threshold bypass
  • Integration tests (store.rs): Verify genesis preservation and finalized head propagation via forkchoice_update
  • Sync tests (full.rs): Verify is_resume_point logic correctly identifies valid resume points (canonical + stateful)

Minor Suggestions

  1. Documentation Clarity (crates/storage/layering.rs:28-48)
    The module-level doc comment explaining the commit gating is excellent and critical for future maintainers.

  2. Dead Code Attributes (crates/storage/layering.rs:108, 242)
    The #[allow(dead_code)] annotations on new and get_commitable_with_threshold are appropriate—retaining the threshold path for regression testing is valuable.

  3. Typo Fix (crates/storage/store.rs:3945)
    The comment change from "writes" to "visits" is correct—RocksDB writes marker files but only visits directories.

Summary

The PR correctly implements finalized-root gating for trie commits, replacing the previous threshold-based approach. This aligns with Ethereum consensus principles (finalized blocks never revert) and prevents potential data loss during reorgs. The code is safe to merge.


Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

Lines of code report

Total lines added: 210
Total lines removed: 0
Total lines changed: 210

Detailed view
+-----------------------------------------+-------+------+
| File                                    | Lines | Diff |
+-----------------------------------------+-------+------+
| ethrex/crates/blockchain/fork_choice.rs | 174   | +4   |
+-----------------------------------------+-------+------+
| ethrex/crates/storage/layering.rs       | 358   | +157 |
+-----------------------------------------+-------+------+
| ethrex/crates/storage/store.rs          | 3333  | +49  |
+-----------------------------------------+-------+------+

@greptile-apps

greptile-apps Bot commented Jun 29, 2026

Copy link
Copy Markdown

Greptile Summary

This PR changes trie-layer persistence to use a canonical safe commit point. The main changes are:

  • Share a safe commit root between forkchoice updates and the trie-layer cache.
  • Remove the old batch-mode commit threshold.
  • Add tests for non-canonical imports and full-sync resume gating.
  • Export the database commit threshold for the reorg-depth compile check.

Confidence Score: 4/5

This is close, but the batch commit path should be fixed before merging.

  • The new gate is based on block depth.
  • Full-sync batch layers can span many blocks.
  • A safe commit root inside a batch may never match a layer boundary.
  • That can leave canonical state unpersisted across restart.

crates/storage/store.rs

Important Files Changed

Filename Overview
crates/storage/store.rs Adds the safe commit root computation and update path after forkchoice changes.
crates/storage/layering.rs Changes commit eligibility to require the shared safe root on the layer parent walk.
crates/blockchain/blockchain.rs Removes the batch-mode flag from update batches.
crates/blockchain/fork_choice.rs Adds a compile-time check tying the database commit threshold to the reorg-depth limit.
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
crates/storage/store.rs:2633
**Batch roots can stall** This computes the commit gate at a block number, but full-sync imports can store one trie layer for a whole batch. When the target block is inside that batch, its state root is not a layer key, so the trie worker walks only batch-boundary parents and never finds the safe root. The batch can stay memory-only, and after a restart the intended resume block can still fail `has_state_root`. The commit point needs to line up with a layer boundary, or the batch path needs metadata that lets interior canonical roots become committable.

Reviews (2): Last reviewed commit: "fix(storage): gate trie-layer commit/pru..." | Re-trigger Greptile

Comment thread crates/storage/store.rs Outdated
Comment thread crates/storage/store.rs Outdated
Comment thread crates/storage/store.rs Outdated
@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Now I have all the verification results. Let me write the final review.


Review: fix(storage): gate trie-layer commit/prune on the finalized head

This PR fixes a real and impactful bug: the in-memory trie layer cache was committing/pruning based on a fixed execution-count threshold (BATCH_COMMIT_THRESHOLD = 4) regardless of canonicality, which allowed non-canonical newPayload blocks to trip the threshold and prune the canonical head's state before any block was finalized — permanently wedging full-sync nodes. The fix replaces the threshold gate with a finalized-root gate: layers are only committed to disk when the CL-reported finalized state root is found on the ancestor walk from the current execution parent. The approach is architecturally sound, the description is clear, and the test suite (unit + integration + p2p) covers the intended behavior well.

Three findings survived verification and are worth addressing:


layering.rs:199 — Step (c) returns Some for already-committed finalized roots

// (c) The executed parent IS the finalized state; commit immediately.
if parent_state_root == finalized {
    return Some(finalized);
}

This short-circuit fires on value equality alone, without checking self.layers.contains_key(&finalized). After a node restart — or after the very first FCU following a genesis/checkpoint sync — forkchoice_update calls set_finalized_state_root(R_k) where R_k is a state already persisted to disk (not present in self.layers). The next apply_trie_updates for block k+1 has parent_state_root = R_k, step (c) fires, get_commitable returns Some(R_k), and apply_trie_updates proceeds to Phase 2: the FKV generator is stopped, a full trie-cache clone is taken, a DB read and write transaction are opened, commit(R_k) silently returns None (empty layers_to_commit), write_tx.commit() commits an empty transaction, the FKV generator is restarted, and the trie cache is RCU'd with an identical copy. No data is corrupted, but real unnecessary work happens on every block built on top of an already-committed finalized root until finalization advances.

Fix: add the layers check to the short-circuit:

if parent_state_root == finalized && self.layers.contains_key(&finalized) {
    return Some(finalized);
}

layering.rs:57, store.rs:67-68commit_threshold is dead configuration on the live path

The commit_threshold field survives in TrieLayerCache, the two constants DB_COMMIT_THRESHOLD = 128 and IN_MEMORY_COMMIT_THRESHOLD = 10000 are still declared in store.rs, and TrieLayerCache::new(commit_threshold) is still a public constructor — but none of these influence any decision on the live commit path. get_commitable (the only live commit gate) reads only self.layers and self.finalized_root; it never touches self.commit_threshold. The constants are threaded through to new_with_finalized and stored but never read.

This creates a concrete maintenance hazard: a Debug print of TrieLayerCache shows commit_threshold: 128, the two constants differ (128 vs 10000), and the constructor accepts the value as a meaningful parameter — all of which imply that tuning these values changes commit frequency when it does not.

Additionally, the #[allow(dead_code)] comment on TrieLayerCache::new at layering.rs:106 says it is "Used via Default::default()…since Default calls it indirectly," but the Default impl at line 88 is a manual struct literal that never calls Self::new(). The new function is genuinely dead in production.

Suggestion: Remove commit_threshold from the struct and both constants from store.rs, or at minimum document clearly that they are test-only artifacts with no effect on production commit behavior.


store.rs:1731, layering.rs:41-47 — Unbounded layer accumulation before first finalization

The removal of BATCH_COMMIT_THRESHOLD (formerly flushed every ~4096 blocks in batch mode) leaves no fallback bound during the pre-finalization window. get_commitable returns None unconditionally while finalized_root == H256::zero(), which persists until the CL sends a non-zero finalizedBlockHash via engine_forkchoiceUpdated. During a full sync from genesis to finalization (~6400 blocks = 2 epochs), all trie diff-layers accumulate in memory with no eviction path. In batch mode this was previously bounded at ~4 batch-layers (each ~1024 blocks); now it is bounded only by available RAM.

The PR acknowledges this tradeoff explicitly ("Pruning non-finalized state to bound memory is unsafe…any future hard bound should pause block intake, not prune. Tracked as a follow-up."), so this is not a surprise — but it represents a real operational regression for full sync that deserves the follow-up work it references.


The core logic — the finalized-root ancestor walk in get_commitable, the RCU pattern, the fcu_lock serialization, and the Arc-sharing between Store and TrieLayerCache — is correct. The regression tests (commit_threshold_layers_no_finalized_must_not_commit, genesis_not_pruned_under_unfinalized_layers, finalized_head_state_retained_after_commit) directly exercise the fixed vs. broken behavior.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@edg-l edg-l changed the title fix(storage): gate trie-layer commit/prune on the finalized head fix(l1): gate trie-layer commit/prune on the finalized head Jun 29, 2026
@github-actions github-actions Bot added the L1 Ethereum client label Jun 29, 2026
@edg-l edg-l marked this pull request as draft June 29, 2026 10:51
@ethrex-project-sync ethrex-project-sync Bot moved this to In Progress in ethrex_l1 Jun 29, 2026
@edg-l edg-l force-pushed the fix/layer-commit-gate-on-finalized branch from 8db39c6 to 31c73c0 Compare June 29, 2026 12:03
@edg-l edg-l changed the title fix(l1): gate trie-layer commit/prune on the finalized head fix(storage): gate trie-layer commit/prune on canonical depth Jun 29, 2026
@edg-l edg-l marked this pull request as ready for review June 29, 2026 12:03
@github-actions github-actions Bot removed the L1 Ethereum client label Jun 29, 2026
@ethrex-project-sync ethrex-project-sync Bot moved this from In Progress to In Review in ethrex_l1 Jun 29, 2026
@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

This PR introduces a canonical+depth commit gate to prevent non-canonical trie layers from being flushed to disk—a critical fix for the "post-state for block 0 is absent" wedge that could corrupt state during reorgs.

Summary of Changes

  • Removes the batch_mode flag and BATCH_COMMIT_THRESHOLD, unifying commit logic under a single safe_commit_root gate.
  • forkchoice_update now computes a canonical safe-commit point (head - DB_COMMIT_THRESHOLD) and shares it with TrieLayerCache via Arc<RwLock<H256>>.
  • get_commitable now walks the layer chain looking for this canonical root; if absent (non-canonical chain or cell is zero), no commit occurs.
  • Adds compile-time assertion ensuring DB_COMMIT_THRESHOLD <= REORG_DEPTH_LIMIT.

Review Feedback

  1. Safety & CorrectnessExcellent. The fix correctly addresses the wedge:

    • The safe_commit_root is only updated in forkchoice_update after canonical tables are updated, ensuring only canonical state roots ever gate commits (store.rs:2586-2590).
    • get_commitable treats a poisoned lock as "not ready" (read().ok()?), which is the conservative/safe choice (layering.rs:147).
    • Cycle guards in the ancestor walk (next == parent_state_root check and steps > max_steps bound) ensure termination even with corrupted layer links (layering.rs:162-172).
  2. Memory Usage During Full SyncAcceptable Trade-off. Removing batch_mode means full sync now uses the 128-layer threshold instead of 4. During batch import (e.g., 1024 blocks), layers accumulate until the next FCU, but the commit of head - 128 after FCU bounds memory to ~128 layers. This is safer than the previous aggressive commit that risked pruning genesis.

  3. Error HandlingGood. set_safe_commit_root returns LockError on poison, and `


Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  1. High: batch/full-sync persistence is broken by the new safe-commit gate. crates/blockchain/blockchain.rs:2641 still sends the whole batch as one UpdateBatch, and crates/storage/store.rs:1532 / crates/storage/store.rs:1560 still turn that into a single trie layer keyed only by the batch’s final child_state_root. But the new gate computes safe_commit_root as the canonical block at head - commit_threshold (crates/storage/store.rs:2624) and get_commitable only fires when that exact root exists on the ancestor layer chain (crates/storage/layering.rs:144, crates/storage/store.rs:3266). In full sync, one layer represents ~1024 blocks, so head - 128 is almost never a batch-boundary root; get_commitable therefore returns None indefinitely. That means batch-imported trie state stops flushing to RocksDB, memory can grow without bound during full sync, and a crash/restart can lose all post-state imported through the batch path. This needs either a batch-aware safe-commit scheme (track batch boundary roots / layer depth) or per-block trie layers in batch mode.

I didn’t find another concrete correctness bug beyond that, but I could not run the test suite here because the environment’s rustup temp directory is read-only.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

Comment thread crates/storage/store.rs
.read()
.map_err(|_| StoreError::LockError)?
.commit_threshold;
let Some(target) = head.checked_sub(commit_threshold as u64) else {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Batch roots can stall This computes the commit gate at a block number, but full-sync imports can store one trie layer for a whole batch. When the target block is inside that batch, its state root is not a layer key, so the trie worker walks only batch-boundary parents and never finds the safe root. The batch can stay memory-only, and after a restart the intended resume block can still fail has_state_root. The commit point needs to line up with a layer boundary, or the batch path needs metadata that lets interior canonical roots become committable.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/storage/store.rs
Line: 2633

Comment:
**Batch roots can stall** This computes the commit gate at a block number, but full-sync imports can store one trie layer for a whole batch. When the target block is inside that batch, its state root is not a layer key, so the trie worker walks only batch-boundary parents and never finds the safe root. The batch can stay memory-only, and after a restart the intended resume block can still fail `has_state_root`. The commit point needs to line up with a layer boundary, or the batch path needs metadata that lets interior canonical roots become committable.

How can I resolve this? If you propose a fix, please make it concise.

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Now I have everything needed for a complete review.


PR #6930 Review: fix(storage): gate trie-layer commit/prune on canonical depth

Overview

This PR fixes a real production wedge where non-canonical newPayload blocks committed trie state to disk via the old depth-only gate, overwriting genesis and stalling full-sync resume. The architectural fix — gating disk commits on a FCU-supplied canonical safe root rather than raw depth — is sound and well-motivated. The compile-time DB_COMMIT_THRESHOLD <= REORG_DEPTH_LIMIT assert and the regression test suite (especially the unit test noncanonical_depth_old_gate_commits_new_gate_does_not) are good engineering. One significant memory regression and one minor logic issue survived verification.


Finding 1 — crates/networking/p2p/sync/full.rs:689 / crates/storage/store.rs:2624 — Full sync accumulates trie layers without bound (memory regression)

Severity: High

During full sync, add_blocks_in_batch processes ~1024 blocks and calls apply_updates once, producing one trie layer keyed by the last block's state_root. forkchoice_update is called once after the batch completes (full.rs:689). At that point compute_safe_commit_root(head) fetches state_root(head - 128) — a block inside the current batch whose intermediate state root was never inserted as a layer key.

When apply_trie_updates then calls get_commitable(parent_state_root) for the batch, the walk starts from the batch's starting state root (on disk, not a layer key), the while-let loop exits immediately, and get_commitable returns None. This repeats for every subsequent batch: safe_commit_root always points to an intra-batch intermediate root that is never a layer key. No disk commit ever fires during full sync.

After K batches (K × ~1024 blocks), K layers live in memory with no upper bound. The old batch_mode=true / BATCH_COMMIT_THRESHOLD=4 path committed aggressively every 4 batch-layers; that bound is now gone with no replacement.

Trigger: any full-sync batch execution where batch_size > DB_COMMIT_THRESHOLD (1024 > 128 — always true in practice).

A fix would be to store per-batch trie diffs as individual per-block layers (so the safe root at head-128 IS a layer key), or to pass the canonical batch-start state root explicitly through the commit gate, or to restore a depth-based fallback for the batch path.


Finding 2 — crates/storage/layering.rs:152 — Check (c) returns Some(safe_root) after safe_root has already been removed from layers

Severity: Low/Medium

Check (c) in get_commitable:

if parent_state_root == safe_root {
    return Some(safe_root);
}

does not verify that safe_root is still present in self.layers. After commit(safe_root) removes it, if forkchoice_update does not advance the canonical head (same head, new payloads still arriving), safe_commit_root retains its old value. Any subsequent apply_trie_updates call whose parent_state_root == safe_root (i.e., a block built directly on the just-committed state root) hits check (c) and returns Some(safe_root). The worker then:

  • sends a Stop to the FKV generator (unnecessary)
  • opens a read + write transaction against RocksDB
  • calls commit(safe_root) which returns None (root gone from layers)
  • unwrap_or_default() at store.rs:3289 gives an empty vec — no corruption
  • sends Continue to FKV generator

This is not a correctness bug (the empty write is safe), but it wastes FKV pause cycles and DB transaction overhead on every trie update while the canonical head is static. The while-let loop in step (d) avoids this naturally: layers.get(parent_state_root) returns None when parent_state_root is the just-committed (removed) root, so it would return None correctly. Check (c) short-circuits before the loop can do the right thing.

Fix: add a membership check in (c): if parent_state_root == safe_root && self.layers.contains_key(&safe_root).


Finding 3 — crates/storage/layering.rs:47 / crates/storage/store.rs:2579 — Unnecessarily wide pub(crate) visibility on the canonical-write invariant

Severity: Low

TrieLayerCache::safe_commit_root is declared pub(crate) but is only read within layering.rs itself (via the shared Arc). Store::set_safe_commit_root and compute_safe_commit_root are pub(crate) but are only called from within store.rs. Any code added to ethrex-storage in the future can call set_safe_commit_root with a non-canonical root or write to the Arc<RwLock<H256>> directly, bypassing the invariant that only forkchoice_update (post-canonicalization) may advance the cell. Given that violating this invariant is exactly the class of bug this PR was written to prevent, tightening visibility is cheap protection.

  • TrieLayerCache::safe_commit_root: drop pub(crate), keep private.
  • Store::set_safe_commit_root, compute_safe_commit_root: drop pub(crate), keep private (or at most pub(super) if tests in the same module need them).

Minor notes (not blocking)

  • layering.rs:160 — the if current == safe_root check inside the while loop fires on the first iteration with current == parent_state_root, but check (c) already returned Some for that case. This first-iteration check is unreachable dead code; it does no harm, but future refactors that remove or move check (c) would silently rely on the loop check without knowing its first iteration is the only time it differs from check (c).

  • layering.rs:165 — the start-of-walk cycle guard (if next == parent_state_root { return None }) is fully subsumed by the bounded-walk guard (steps > max_steps). It is a performance optimization (exits one step earlier for simple cycles), but the comment describing both as covering disjoint cases is slightly misleading.

  • store.rs:2594 — if set_safe_commit_root returns LockError (poisoned), forkchoice_update returns an error to the caller even though forkchoice_update_inner already succeeded and the canonical DB is fully updated. A caller that retries on error would perform a double-FCU on already-canonical state. The comment acknowledges this is intended as unrecoverable, but it may be worth logging a clear panic/fatal rather than surfacing a retriable error.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@edg-l edg-l changed the title fix(storage): gate trie-layer commit/prune on canonical depth fix(l1): gate trie-layer commit/prune on canonical depth Jun 30, 2026
@github-actions github-actions Bot added the L1 Ethereum client label Jun 30, 2026
Full-sync ethrex nodes could wedge permanently, looping "Full sync cannot
resume: post-state for block 0 is absent ... resume_parent_number=0
local_head=0". Under a stale consensus forkchoice, non-canonical newPayload
blocks executed ahead of canonicalization tripped the fixed-depth layer-commit
threshold, flushing non-canonical state to the single path-keyed on-disk root
and overwriting genesis; since those blocks never canonicalized, the full-sync
resume gate (is_canonical_sync && has_state_root) found no canonical+stateful
block and paused forever.

Gate trie diff-layer disk commits on CANONICAL + DEPTH: the Store computes
safe_commit_root = state_root of the canonical block at head - commit_threshold
(in forkchoice_update, post-canonicalization) and threads it through a dedicated
Arc<RwLock<H256>> cell shared with TrieLayerCache. The worker's get_commitable
returns that root only if found on the executed parent walk; otherwise nothing
is committed. Non-canonical state is never flushed (genesis preserved), memory
stays bounded to ~commit_threshold layers regardless of finalization, and
commits resume deterministically without depending on finalization.

A compile-time assert ties DB_COMMIT_THRESHOLD to REORG_DEPTH_LIMIT so a
committed block can never be reorged out.

Adds white-box gate tests (incl. a red->green regression contrasting the old
depth-only gate vs the new gate), a public-API integration test, and relocates
the is_resume_point tests to the ethrex-test crate.
@edg-l edg-l force-pushed the fix/layer-commit-gate-on-finalized branch from edf7615 to 95befe5 Compare July 1, 2026 08:36
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

⚠️ Known Issues — intentionally skipped tests

Source: docs/known_issues.md

Stateless blockchain EF-tests (zkevm bundle) skipped under Amsterdam v6.1.0

make -C tooling/ef_tests/blockchain test-stateless runs against the
tests-zkevm@v0.4.1 fixture bundle — currently the only published zkevm test
release. Those fixtures were filled against an older glamsterdam devnet and
re-execute every case under the for_amsterdam fork, so they lag the
glamsterdam-devnet v6.1.0 gas accounting this client now implements
(EIP-8037 / EIP-8038 / EIP-2780 / EIP-7976 / EIP-7981 …). Re-executing them
yields ~2790/2864 stale-gas failures ("Transaction execution unexpectedly
failed"), spread pervasively across every fork and through the EIP-8025 proof
suite, so there is no clean passing subset to keep.

Until a v6.1.0-aligned zkevm bundle is published, the entire bundle is skipped
for the stateless run via the fork_Amsterdam entry in the stateless-only
EXTRA_SKIPS (tooling/ef_tests/blockchain/tests/all.rs) — every test in this
Amsterdam-only bundle carries the [fork_Amsterdam-…] parametrization in its
test key. The skip is #[cfg(feature = "stateless")], so it does not touch
the non-stateless test-levm run. Coverage of these EIPs is retained by
test-levm, the engine EF-tests, and the state EF-tests, all of which execute
against the live v6.1.0 fixtures and pass.

Re-enable by removing the "fork_Amsterdam" skip once .fixtures_url_zkevm
points at a zkevm bundle filled for glamsterdam-devnet v6.1.0.

Commit-to-disk (Phase 2 of the trie-update worker) only ran while blocks
were executing, and forkchoice_update merely advanced safe_commit_root
without poking the worker. An execute-all-then-one-forkchoice flow (block
import) therefore accumulated every layer and never persisted: the disk
stayed frozen at genesis, so the snap server kept serving genesis state
that should be unavailable (>127 blocks old) — failing hive devp2p
snap/AccountRange Test 11.

Add TrieMessage::Commit(root); forkchoice_update sends it when the
safe-commit root advances, flushing the committable backlog up to it.
Phase 2 is extracted into commit_to_disk and shared by both paths.
RocksDB-backed (DB_COMMIT_THRESHOLD=128) test: import >128 blocks via
add_block then forkchoice_update, and assert the now-canonical backlog is
flushed so the genesis state root is no longer serveable while head state
remains. Mirrors hive devp2p snap/AccountRange Test 11. Extracts a shared
load_funded_genesis helper from setup_store.
@edg-l edg-l force-pushed the fix/layer-commit-gate-on-finalized branch from 95befe5 to e03304f Compare July 1, 2026 09:06
Comment thread crates/storage/store.rs
/// Whether this batch comes from full sync (batch execution mode). When true,
/// the persist worker waits for the disk flush (`wait_for_flush`) to bound
/// in-flight batches during bulk import.
pub batch_mode: bool,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batch full-sync question. In batch mode, apply_updates registers ONE layer per UpdateBatch keyed by the end-of-batch last_state_root (spans ~1024 blocks). compute_safe_commit_root(head) returns the state root of the canonical block at head - commit_threshold = head - 128 — that's per-block granularity, so it rarely lands on a batch boundary. The returned root is then not a layer key, get_commitable walks past it, and returns None. Iterating that: batch 1 finishes at block 1024, FCU sets safe_commit to block 896's state_root (mid-batch, no layer) → None; batch 2 to block 1920 → still no layer → None; and so on. Layers accumulate at ~1 GB per batch — up to ~128 layers before any pressure kicks in — whereas the removed BATCH_COMMIT_THRESHOLD = 4 capped this at ~4 batches.

The description says "Memory is bounded to ~commit_threshold layers in all cases", so I'd like to sanity-check my read. Is there a mechanism that snaps safe_commit_root to a batch boundary in batch mode (e.g., the full-sync driver issues FCU only at block numbers where head-128 is on a boundary), or is the batch-mode cadence something that will be looked at in follow-up? Not blocking — just want to make sure I understand the intended behavior.

…-on-finalized

# Conflicts:
#	crates/storage/store.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Glamsterdam L1 Ethereum client

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

3 participants