Skip to content

feat(l1): implement debug_getBadBlocks with bad-block tracking#6948

Open
edg-l wants to merge 1 commit into
mainfrom
feat/debug-get-bad-blocks-impl
Open

feat(l1): implement debug_getBadBlocks with bad-block tracking#6948
edg-l wants to merge 1 commit into
mainfrom
feat/debug-get-bad-blocks-impl

Conversation

@edg-l

@edg-l edg-l commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Description

Implements debug_getBadBlocks with real bad-block tracking, superseding #6693 (which returned a hardcoded empty array as a placeholder).

Blocks rejected during engine_newPayload execution are now persisted and returned by the RPC method, matching geth's BadBlockArgs response shape.

Changes

Storage (ethrex-storage)

  • New BAD_BLOCKS column family storing a single-keyed, RLP-encoded Vec<Block> list.
  • Store::add_bad_block — dedups on (number, hash), sorts by descending block number, and caps the list at 16 entries (oldest evicted).
  • Store::get_bad_blocks — returns the tracked list (empty when unset).

Population (engine/payload.rs)

  • The block is recorded at the two newPayload INVALID decision points (InvalidBlock, EvmError), alongside the existing set_latest_valid_ancestor call. These already exclude ParentNotFound / ParentStateNotFound, which are SYNCING rather than bad.

RPC (debug/bad_blocks.rs)

  • GetBadBlocksRequest returns [{ hash, block, rlp }] per entry: the full block JSON (via RpcBlock) plus the 0x-hex block RLP. Routed as debug_getBadBlocks under the existing debug namespace allowlist.

Testing

  • test/tests/storage/store_tests.rs::test_bad_blocks covers the empty default, descending ordering, (number, hash) dedup, and the 16-entry cap with oldest eviction (runs under both in-memory and rocksdb engines).

Scope note

Bad-block recording is wired into the engine_newPayload path — the canonical "bad blocks seen on the network" source per the execution-apis spec. The full-sync batch/pipeline import paths also reject blocks but would require per-block clones on hot loops to record; left out of scope for this change.

Closes #6693

@edg-l edg-l requested a review from a team as a code owner July 2, 2026 11:33
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

⚠️ Known Issues — intentionally skipped tests

Source: docs/known_issues.md

Stateless blockchain EF-tests (zkevm bundle) skipped under Amsterdam v6.1.0

make -C tooling/ef_tests/blockchain test-stateless runs against the
tests-zkevm@v0.4.1 fixture bundle — currently the only published zkevm test
release. Those fixtures were filled against an older glamsterdam devnet and
re-execute every case under the for_amsterdam fork, so they lag the
glamsterdam-devnet v6.1.0 gas accounting this client now implements
(EIP-8037 / EIP-8038 / EIP-2780 / EIP-7976 / EIP-7981 …). Re-executing them
yields ~2790/2864 stale-gas failures ("Transaction execution unexpectedly
failed"), spread pervasively across every fork and through the EIP-8025 proof
suite, so there is no clean passing subset to keep.

Until a v6.1.0-aligned zkevm bundle is published, the entire bundle is skipped
for the stateless run via the fork_Amsterdam entry in the stateless-only
EXTRA_SKIPS (tooling/ef_tests/blockchain/tests/all.rs) — every test in this
Amsterdam-only bundle carries the [fork_Amsterdam-…] parametrization in its
test key. The skip is #[cfg(feature = "stateless")], so it does not touch
the non-stateless test-levm run. Coverage of these EIPs is retained by
test-levm, the engine EF-tests, and the state EF-tests, all of which execute
against the live v6.1.0 fixtures and pass.

Re-enable by removing the "fork_Amsterdam" skip once .fixtures_url_zkevm
points at a zkevm bundle filled for glamsterdam-devnet v6.1.0.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Kimi Code Review

Overall this is a solid implementation of debug_getBadBlocks. The bounded list approach (max 16) prevents DoS via storage exhaustion, and the deduplication by (number, hash) correctly handles re-orgs.

Issues to address:

1. Race condition in add_bad_block (Store)
crates/storage/store.rs:1328-1338 uses a non-atomic read-modify-write pattern. Concurrent calls to add_bad_block can lose updates:

// Current (racy):
let mut bad_blocks = self.get_bad_blocks().await?;  // Read
bad_blocks.push(block);                              // Modify
// ... other thread modifies here ...
self.write_async(..., bad_blocks.encode_to_vec()).await  // Write (overwrites other thread)

Since this is a debug feature with low concurrency, you may accept this risk, but document it with a // FIXME: racy read-modify-write comment, or wrap in a storage transaction if your backend supports it.

2. Potential panic/error on RpcBlock::build with receipts
crates/networking/rpc/debug/bad_blocks.rs:29 passes true for include_tx_receipts:

let rpc_block = RpcBlock::build(block.header, block.body, hash, true)?;

For invalid blocks, transaction receipts may not exist in storage (execution failed). If RpcBlock::build queries the database for receipts, this will return an error or panic for bad blocks. Consider passing false unless you've verified that RpcBlock handles missing receipts gracefully for non-canonical blocks.

3. Unnecessary clone on happy path
crates/networking/rpc/engine/payload.rs:1245 clones the block unconditionally:

let bad_block_candidate = block.clone();

While the comment notes this is "negligible", for a 30M gas block with full blobs, this could be ~100MB+ of memory copying on every newPayload call, even when the block is valid. Consider using Arc<Block> in the add_block signature if possible, or documenting the memory overhead. If you keep the clone, ensure it's truly negligible in benchmarks.

Minor suggestions:

  • Documentation: In crates/storage/api/tables.rs:116-119, clarify that duplicates are ignored in the comment (currently only mentions sorting and bounding).

  • Test coverage: The test in store_tests.rs verifies the logic well, but add a test case for concurrent add_bad_block calls to document the expected behavior (even if it's racy).

Security check:

  • No unbounded storage growth (✓ bounded to 16)
  • No sensitive data exposure (✓ only stores already-seen block data)
  • RLP encoding uses standard library (✓)

Verdict: Fix the include_tx_receipts concern before merging. The race condition is acceptable for a debug feature if documented.


Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

  1. payload.rs and payload.rs: add_bad_block(...).await? is now on the INVALID path, so any failure in bad-block bookkeeping turns a known-invalid payload into a JSON-RPC internal error instead of returning PayloadStatus::invalid. That can happen from a decode failure in store.rs, a write failure, or corrupted BAD_BLOCKS contents. debug_getBadBlocks should not be able to change Engine API semantics; this write needs to be best-effort or at least isolated from the response path.

  2. store.rs: add_bad_block is a read-modify-write over a single key with no synchronization. Two concurrent invalid newPayload requests can both read the same old vector, append different blocks, and the later write drops the earlier one. The block executor channel serializes execution, but these RPC tasks resume independently after the await, so this race is still reachable. This needs a mutex/CAS/merge-style update if the list is meant to be reliable.

  3. payload.rs, error.rs, and errors.rs: the new persistence also records every ChainError::EvmError as a “bad block”, but that bucket still includes local DB and Custom failures, not just deterministic consensus-invalid payloads. That means debug_getBadBlocks can report locally failed blocks as if they were objectively bad. If the goal is to mirror geth-style bad-block debugging, I’d restrict persistence to InvalidBlock or to a narrower subset of deterministic validation failures.

I did not run the test suite: cargo test tried to update the Rust toolchain and failed because rustup could not write under /home/runner/.rustup in this environment.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

Lines of code report

Total lines added: 67
Total lines removed: 0
Total lines changed: 67

Detailed view
+--------------------------------------------------+-------+------+
| File                                             | Lines | Diff |
+--------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/debug/bad_blocks.rs | 34    | +34  |
+--------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/debug/mod.rs        | 4     | +1   |
+--------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/engine/payload.rs   | 1378  | +3   |
+--------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/rpc.rs              | 1382  | +2   |
+--------------------------------------------------+-------+------+
| ethrex/crates/storage/api/tables.rs              | 25    | +1   |
+--------------------------------------------------+-------+------+
| ethrex/crates/storage/store.rs                   | 3264  | +26  |
+--------------------------------------------------+-------+------+

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

That's fine, no cleanup needed since the fetch never succeeded. Here's the review.

Review: debug_getBadBlocks implementation (PR 6948)

Overall a clean, well-scoped feature with good test coverage of the bounding/dedup/ordering logic. Two issues worth addressing before merge, plus a few smaller notes.

1. Unconditional block clone on every newPayload call (performance)

crates/networking/rpc/engine/payload.rs:1245

let bad_block_candidate = block.clone();
match add_block(context, block, bal, make_witness).await {

The clone runs on every try_execute_payload invocation, including the overwhelming majority that succeed — not just the error paths that actually need it. For blocks with many transactions this is a full deep clone (header + body + all transactions) done purely speculatively, on the hot path of block import. The comment claims it's "negligible next to block execution," but that's asserted, not measured, and it's wasted allocation/copy work in the common case.

Consider restructuring so the clone only happens once you're already on an error path — e.g. have add_block return the consumed block alongside the error (Err((ChainError, Block))), or wrap the incoming block in Arc<Block> earlier in the call chain so the "keep a copy" step is an Arc::clone instead of a deep clone.

2. Non-atomic read-modify-write race in Store::add_bad_block

crates/storage/store.rs:1318-1341

pub async fn add_bad_block(&self, block: Block) -> Result<(), StoreError> {
    let mut bad_blocks = self.get_bad_blocks().await?;   // separate read txn
    ...
    self.write_async(BAD_BLOCKS, BAD_BLOCKS_KEY.to_vec(), bad_blocks.encode_to_vec()).await  // separate write txn
}

get_bad_blocks/write_async are independent transactions (confirmed in read_async/write_async, store.rs:1468-1500), and nothing in try_execute_payload or Blockchain serializes concurrent newPayload executions. If two invalid blocks are processed concurrently (plausible if a CL or multiple RPC callers issue overlapping engine_newPayload requests, e.g. around a reorg), this is a classic lost-update: both read the same base list, and the second writer's commit() clobbers the first writer's recorded bad block. The consequence is only a debug-visibility gap (not consensus-affecting), but since the whole point of this feature is faithfully recording bad blocks, silently dropping one under concurrency undercuts it. Worth wrapping the read+modify+write in a single transaction/lock (or serializing via a Mutex around this method) to make it atomic.

3. ChainError::EvmError includes internal DB errors, not just consensus violations (design note, pre-existing branch)

crates/vm/errors.rs:10EvmError::DB(String) is produced from LevmDatabaseError/InternalError::Database, i.e. a local storage read failure, not necessarily a block that violates consensus rules. try_execute_payload already treated ChainError::EvmError as INVALID before this PR (payload.rs:1274-1285), so this isn't a regression introduced here — but this PR adds a new consequence: such blocks are now permanently persisted to the BAD_BLOCKS table and surfaced over debug_getBadBlocks. A transient local DB hiccup could now show up as a falsely-accused "bad block" in the debug API, which is misleading for anyone diagnosing the node using this endpoint. Worth at least a comment/mental-note that this endpoint conflates "provably invalid per consensus rules" with "this node failed to execute it," matching geth's own known looseness here, or scoping add_bad_block to just InvalidBlock if a cleaner signal is preferred.

Minor observations

  • bad_blocks.rs:31 — RLP re-encodes the block for the rlp field, and hash() (payload.rs / bad_blocks.rs) is recomputed independently from the block already held; both are cheap here since the list is capped at 16, no action needed.
  • The BAD_BLOCKS table stores the whole bounded list under a single key (store.rs:1330-1337), so every insert rewrites all ≤16 full blocks. Given the hard 16-entry cap this is bounded and acceptable, but note it as a place that could grow expensive if MAX_BAD_BLOCKS is ever raised significantly.
  • New column family correctly integrates with the RocksDB open path (rocksdb.rs:88-93 unions existing CFs with TABLES), so no migration/upgrade concern there.
  • Test coverage (test_bad_blocks) is solid — covers empty default, descending sort, dedup, and eviction bound across both storage engines.

Automated review by Claude (Anthropic) · sonnet · custom prompt

@greptile-apps

greptile-apps Bot commented Jul 2, 2026

Copy link
Copy Markdown

Greptile Summary

This PR implements debug_getBadBlocks with real bad-block tracking, replacing the earlier placeholder that returned a hardcoded empty array. Blocks rejected by engine_newPayload as InvalidBlock or EvmError are now persisted in a new BAD_BLOCKS RocksDB column family and returned by the RPC method in geth's BadBlockArgs shape.

  • Storage layer (store.rs, tables.rs): A single-keyed RLP-encoded Vec<Block> is maintained under a new BAD_BLOCKS column family; add_bad_block deduplicates on hash, sorts descending by block number, and caps the list at 16 entries (lowest-numbered evicted when full).
  • Engine path (engine/payload.rs): The block is cloned before add_block consumes it, then recorded at both INVALID arm exits; SYNCING arms (ParentNotFound, ParentStateNotFound) are correctly left unrecorded.
  • RPC handler (debug/bad_blocks.rs, rpc.rs): GetBadBlocksRequest returns [{ hash, block, rlp }] per entry and is routed under the existing debug namespace.

Confidence Score: 4/5

The change is well-scoped and additive; the only correctness concern is a read-modify-write in add_bad_block that can drop one entry under concurrent invalid payloads, which is tolerable for a debug list.

The core engine path and RPC handler are correct. The add_bad_block storage function uses an unguarded read-modify-write pattern that can cause a bad block entry to be silently lost when two newPayload failures race — negligible impact for a debug endpoint but worth addressing before this is relied on for diagnostics.

crates/storage/store.rs — the add_bad_block read-modify-write and the redundant number+hash dedup predicate.

Important Files Changed

Filename Overview
crates/networking/rpc/debug/bad_blocks.rs New RPC handler for debug_getBadBlocks; correctly reads stored bad blocks and serializes them to the geth BadBlockArgs shape with hash, full-transaction RPC block, and 0x-hex RLP.
crates/networking/rpc/engine/payload.rs Clones the block before add_block consumes it and records it on the two INVALID decision points (InvalidBlock, EvmError); ParentNotFound and ParentStateNotFound (SYNCING) are correctly left unrecorded.
crates/storage/store.rs Adds add_bad_block / get_bad_blocks with RLP-encoded bounded list; has an unguarded read-modify-write and a redundant number+hash dedup predicate, but is otherwise correct.
crates/storage/api/tables.rs Adds BAD_BLOCKS column family and bumps the TABLES array from 20 to 21 entries.
test/tests/storage/store_tests.rs Adds test_bad_blocks covering the empty default, descending-order sort, hash dedup, and 16-entry cap with oldest eviction; runs under both in-memory and RocksDB engines.
crates/networking/rpc/rpc.rs Routes debug_getBadBlocks to the new GetBadBlocksRequest handler in the existing debug namespace.
crates/networking/rpc/debug/mod.rs Exposes the new bad_blocks module.
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
crates/storage/store.rs:1325-1344
**Unguarded read-modify-write on concurrent `newPayload` calls**

`add_bad_block` reads the list, mutates it in memory, and writes it back with no atomic protection. If two concurrent `newPayload` requests both fail (e.g. two different peers submit different invalid blocks at the same time), the second writer overwrites the first writer's appended entry and one bad block is silently lost from the list. For a bounded debug list this is low-impact, but worth acknowledging or documenting as a known limitation.

### Issue 2 of 2
crates/storage/store.rs:1329-1332
The dedup check combines `block_number` and `block_hash`, but block number is already embedded in the block header that the hash is derived from — two blocks with identical hashes necessarily have identical numbers. Deduplicating on `block_hash` alone is sufficient and clearer.

```suggestion
        if bad_blocks.iter().any(|b| b.hash() == block_hash) {
```

Reviews (1): Last reviewed commit: "feat(l1): implement debug_getBadBlocks R..." | Re-trigger Greptile

Comment thread crates/storage/store.rs
Comment on lines +1325 to +1344
pub async fn add_bad_block(&self, block: Block) -> Result<(), StoreError> {
let mut bad_blocks = self.get_bad_blocks().await?;
let block_number = block.header.number;
let block_hash = block.hash();
if bad_blocks
.iter()
.any(|b| b.header.number == block_number && b.hash() == block_hash)
{
return Ok(());
}
bad_blocks.push(block);
bad_blocks.sort_by(|a, b| b.header.number.cmp(&a.header.number));
bad_blocks.truncate(MAX_BAD_BLOCKS);
self.write_async(
BAD_BLOCKS,
BAD_BLOCKS_KEY.to_vec(),
bad_blocks.encode_to_vec(),
)
.await
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unguarded read-modify-write on concurrent newPayload calls

add_bad_block reads the list, mutates it in memory, and writes it back with no atomic protection. If two concurrent newPayload requests both fail (e.g. two different peers submit different invalid blocks at the same time), the second writer overwrites the first writer's appended entry and one bad block is silently lost from the list. For a bounded debug list this is low-impact, but worth acknowledging or documenting as a known limitation.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/storage/store.rs
Line: 1325-1344

Comment:
**Unguarded read-modify-write on concurrent `newPayload` calls**

`add_bad_block` reads the list, mutates it in memory, and writes it back with no atomic protection. If two concurrent `newPayload` requests both fail (e.g. two different peers submit different invalid blocks at the same time), the second writer overwrites the first writer's appended entry and one bad block is silently lost from the list. For a bounded debug list this is low-impact, but worth acknowledging or documenting as a known limitation.

How can I resolve this? If you propose a fix, please make it concise.

Comment thread crates/storage/store.rs
Comment on lines +1329 to +1332
if bad_blocks
.iter()
.any(|b| b.header.number == block_number && b.hash() == block_hash)
{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The dedup check combines block_number and block_hash, but block number is already embedded in the block header that the hash is derived from — two blocks with identical hashes necessarily have identical numbers. Deduplicating on block_hash alone is sufficient and clearer.

Suggested change
if bad_blocks
.iter()
.any(|b| b.header.number == block_number && b.hash() == block_hash)
{
if bad_blocks.iter().any(|b| b.hash() == block_hash) {
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/storage/store.rs
Line: 1329-1332

Comment:
The dedup check combines `block_number` and `block_hash`, but block number is already embedded in the block header that the hash is derived from — two blocks with identical hashes necessarily have identical numbers. Deduplicating on `block_hash` alone is sufficient and clearer.

```suggestion
        if bad_blocks.iter().any(|b| b.hash() == block_hash) {
```

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread crates/storage/store.rs
/// `debug_getBadBlocks`. The list is bounded to [`MAX_BAD_BLOCKS`] entries,
/// kept sorted by descending block number, with the oldest dropped once the
/// bound is exceeded. Duplicate `(number, hash)` entries are ignored.
pub async fn add_bad_block(&self, block: Block) -> Result<(), StoreError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: non-atomic so it could lose an entry
In practice it won't happen (bad blocks are rare, simultaneously executing two blocks is also rare even on fast networks like bsc).

Comment thread crates/storage/store.rs
}
bad_blocks.push(block);
bad_blocks.sort_by(|a, b| b.header.number.cmp(&a.header.number));
bad_blocks.truncate(MAX_BAD_BLOCKS);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eviction semantic worth double-checking against geth. This truncates by descending block number (keeps the 16 highest-numbered bad blocks), whereas geth's bad-block cache is an LRU keyed by insertion time (returns "most recently seen"). Divergence only shows up when a bad block at a lower number arrives after the 16 slots are full — e.g., a losing-fork bad block reaching us after a deep reorg. Under this code it's silently dropped and never appears in debug_getBadBlocks; under geth it enters at the head of the LRU. If tooling that consumes the endpoint assumes "most recent by clock" semantics, the mismatch could confuse it. Might be intentional (highest-number is arguably a cleaner proxy for "newest fork"); worth pinning either the semantic choice in a doc comment or the divergence-vs-geth in the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

3 participants