fix(l1): gate trie-layer commit/prune on canonical depth#6930
Conversation
🤖 Codex Code ReviewFindings:
Open question:
I couldn’t run the targeted Rust tests here because Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Kimi Code ReviewThis PR replaces the threshold-based trie commit strategy with a finalized-root gate, ensuring non-finalized state is never persisted to disk (protecting against reorg data loss). The implementation is correct and well-tested. Code Quality & Correctness
Security & Consensus
Testing The test coverage is comprehensive:
Minor Suggestions
Summary The PR correctly implements finalized-root gating for trie commits, replacing the previous threshold-based approach. This aligns with Ethereum consensus principles (finalized blocks never revert) and prevents potential data loss during reorgs. The code is safe to merge. Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
Lines of code reportTotal lines added: Detailed view |
Greptile SummaryThis PR changes trie-layer persistence to use a canonical safe commit point. The main changes are:
Confidence Score: 4/5This is close, but the batch commit path should be fixed before merging.
crates/storage/store.rs
|
| Filename | Overview |
|---|---|
| crates/storage/store.rs | Adds the safe commit root computation and update path after forkchoice changes. |
| crates/storage/layering.rs | Changes commit eligibility to require the shared safe root on the layer parent walk. |
| crates/blockchain/blockchain.rs | Removes the batch-mode flag from update batches. |
| crates/blockchain/fork_choice.rs | Adds a compile-time check tying the database commit threshold to the reorg-depth limit. |
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
crates/storage/store.rs:2633
**Batch roots can stall** This computes the commit gate at a block number, but full-sync imports can store one trie layer for a whole batch. When the target block is inside that batch, its state root is not a layer key, so the trie worker walks only batch-boundary parents and never finds the safe root. The batch can stay memory-only, and after a restart the intended resume block can still fail `has_state_root`. The commit point needs to line up with a layer boundary, or the batch path needs metadata that lets interior canonical roots become committable.
Reviews (2): Last reviewed commit: "fix(storage): gate trie-layer commit/pru..." | Re-trigger Greptile
🤖 Claude Code ReviewNow I have all the verification results. Let me write the final review. Review:
|
8db39c6 to
31c73c0
Compare
🤖 Kimi Code ReviewThis PR introduces a canonical+depth commit gate to prevent non-canonical trie layers from being flushed to disk—a critical fix for the "post-state for block 0 is absent" wedge that could corrupt state during reorgs. Summary of Changes
Review Feedback
Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code Review
I didn’t find another concrete correctness bug beyond that, but I could not run the test suite here because the environment’s Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
| .read() | ||
| .map_err(|_| StoreError::LockError)? | ||
| .commit_threshold; | ||
| let Some(target) = head.checked_sub(commit_threshold as u64) else { |
There was a problem hiding this comment.
Batch roots can stall This computes the commit gate at a block number, but full-sync imports can store one trie layer for a whole batch. When the target block is inside that batch, its state root is not a layer key, so the trie worker walks only batch-boundary parents and never finds the safe root. The batch can stay memory-only, and after a restart the intended resume block can still fail
has_state_root. The commit point needs to line up with a layer boundary, or the batch path needs metadata that lets interior canonical roots become committable.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/storage/store.rs
Line: 2633
Comment:
**Batch roots can stall** This computes the commit gate at a block number, but full-sync imports can store one trie layer for a whole batch. When the target block is inside that batch, its state root is not a layer key, so the trie worker walks only batch-boundary parents and never finds the safe root. The batch can stay memory-only, and after a restart the intended resume block can still fail `has_state_root`. The commit point needs to line up with a layer boundary, or the batch path needs metadata that lets interior canonical roots become committable.
How can I resolve this? If you propose a fix, please make it concise.
🤖 Claude Code ReviewNow I have everything needed for a complete review. PR #6930 Review:
|
Full-sync ethrex nodes could wedge permanently, looping "Full sync cannot resume: post-state for block 0 is absent ... resume_parent_number=0 local_head=0". Under a stale consensus forkchoice, non-canonical newPayload blocks executed ahead of canonicalization tripped the fixed-depth layer-commit threshold, flushing non-canonical state to the single path-keyed on-disk root and overwriting genesis; since those blocks never canonicalized, the full-sync resume gate (is_canonical_sync && has_state_root) found no canonical+stateful block and paused forever. Gate trie diff-layer disk commits on CANONICAL + DEPTH: the Store computes safe_commit_root = state_root of the canonical block at head - commit_threshold (in forkchoice_update, post-canonicalization) and threads it through a dedicated Arc<RwLock<H256>> cell shared with TrieLayerCache. The worker's get_commitable returns that root only if found on the executed parent walk; otherwise nothing is committed. Non-canonical state is never flushed (genesis preserved), memory stays bounded to ~commit_threshold layers regardless of finalization, and commits resume deterministically without depending on finalization. A compile-time assert ties DB_COMMIT_THRESHOLD to REORG_DEPTH_LIMIT so a committed block can never be reorged out. Adds white-box gate tests (incl. a red->green regression contrasting the old depth-only gate vs the new gate), a public-API integration test, and relocates the is_resume_point tests to the ethrex-test crate.
edf7615 to
95befe5
Compare
|
Commit-to-disk (Phase 2 of the trie-update worker) only ran while blocks were executing, and forkchoice_update merely advanced safe_commit_root without poking the worker. An execute-all-then-one-forkchoice flow (block import) therefore accumulated every layer and never persisted: the disk stayed frozen at genesis, so the snap server kept serving genesis state that should be unavailable (>127 blocks old) — failing hive devp2p snap/AccountRange Test 11. Add TrieMessage::Commit(root); forkchoice_update sends it when the safe-commit root advances, flushing the committable backlog up to it. Phase 2 is extracted into commit_to_disk and shared by both paths.
RocksDB-backed (DB_COMMIT_THRESHOLD=128) test: import >128 blocks via add_block then forkchoice_update, and assert the now-canonical backlog is flushed so the genesis state root is no longer serveable while head state remains. Mirrors hive devp2p snap/AccountRange Test 11. Extracts a shared load_funded_genesis helper from setup_store.
95befe5 to
e03304f
Compare
| /// Whether this batch comes from full sync (batch execution mode). When true, | ||
| /// the persist worker waits for the disk flush (`wait_for_flush`) to bound | ||
| /// in-flight batches during bulk import. | ||
| pub batch_mode: bool, |
There was a problem hiding this comment.
Batch full-sync question. In batch mode, apply_updates registers ONE layer per UpdateBatch keyed by the end-of-batch last_state_root (spans ~1024 blocks). compute_safe_commit_root(head) returns the state root of the canonical block at head - commit_threshold = head - 128 — that's per-block granularity, so it rarely lands on a batch boundary. The returned root is then not a layer key, get_commitable walks past it, and returns None. Iterating that: batch 1 finishes at block 1024, FCU sets safe_commit to block 896's state_root (mid-batch, no layer) → None; batch 2 to block 1920 → still no layer → None; and so on. Layers accumulate at ~1 GB per batch — up to ~128 layers before any pressure kicks in — whereas the removed BATCH_COMMIT_THRESHOLD = 4 capped this at ~4 batches.
The description says "Memory is bounded to ~commit_threshold layers in all cases", so I'd like to sanity-check my read. Is there a mechanism that snaps safe_commit_root to a batch boundary in batch mode (e.g., the full-sync driver issues FCU only at block numbers where head-128 is on a boundary), or is the batch-mode cadence something that will be looked at in follow-up? Not blocking — just want to make sure I understand the intended behavior.
…-on-finalized # Conflicts: # crates/storage/store.rs
Problem
Full-sync ethrex nodes can wedge permanently, looping:
Observed on glamsterdam-devnet-6 (prysm-paired ethrex nodes). This is the Q2 half of the incident cluster — see #6921 (trigger: peers not serving headers by hash) and #6924 (Q1: syncer lock held while waiting for a forkchoice head). Q1 (the lock) was masking Q2 (this state-resume wedge); once the syncer isn't pinned, the node actively cycles and hits this.
Root cause: trie diff-layers were committed/pruned by a fixed-depth count (
DB_COMMIT_THRESHOLD) regardless of canonicality. Under a stale consensus forkchoice, non-canonicalnewPayloadblocks executed ahead of canonicalization tripped that threshold, flushing non-canonical state to the single path-keyed on-disk root and overwriting genesis. Since those blocks never canonicalized (local_headstayed 0), the full-sync resume gate (is_resume_point = is_canonical_sync && has_state_root) found no canonical+stateful block and paused every cycle.Fix
Gate trie diff-layer disk commits on CANONICAL + DEPTH rather than depth alone:
forkchoice_update(after canonicalization), the Store computessafe_commit_root= the state_root of the canonical block athead - commit_threshold, and writes it into a dedicatedArc<RwLock<H256>>cell shared withTrieLayerCache.get_commitable(parent_state_root)returns that root only if it is found on the executed parent walk; otherwise it commits nothing.H256::zero()(chain shorter than the threshold, e.g. genesis init) → no commit, so genesis-on-disk is never gated away.std::RwLockguard held across.await.Consequences:
commit_thresholdlayers in all cases, including indefinite non-finality (no dependence on finalization — distinct from a finalized-gated approach, which would grow unbounded on devnets / finality incidents).const _: ()assert tiesDB_COMMIT_THRESHOLDtoREORG_DEPTH_LIMITso a committed block can never be reorged out.Removes the now-unused
BATCH_COMMIT_THRESHOLD/is_batch/batch_modedepth-only commit path.Testing
layering.rs, including a red→green regression (noncanonical_depth_old_gate_commits_new_gate_does_not) contrasting the old depth-only gate (commits non-canonical state) against the new gate (commits nothing while no canonical safe-commit point exists), plus a memory-bound assertion.test/tests/blockchain/canonical_commit_gate_tests.rs: non-canonical imports (no FCU) never prune genesis.is_resume_pointtests relocated totest/tests/p2p/.cargo test --workspace --exclude 'ethrex-l2*' --exclude ethrex-prover --exclude ethrex-guest-program(green),make -C tooling/ef_tests/blockchain test(green — 0 "state root missing for block 0"),make -C tooling/ef_tests/engine test(green),cargo clippy --workspace … -D warnings(clean).Relationship to #6924 / #6921
Complementary, different files (this PR:
storage/blockchain; #6924:sync_manager.rs). #6924's description left this exact "Full sync cannot resume …removedb" case as a separate change, suggesting an automatic reset-and-resync; this PR takes the preventive angle (never strand the resume gate in the first place), which should make a recovery mechanism unnecessary for this cause. Worth aligning on the Q2 direction.