fix: honour engine fixture errorCode instead of accepting any RPC error - #12783
Draft
Marchhill wants to merge 1 commit into
Draft
fix: honour engine fixture errorCode instead of accepting any RPC error#12783Marchhill wants to merge 1 commit into
Marchhill wants to merge 1 commit into
Conversation
AssertExpectedRpcError treated any JSON-RPC error from engine_newPayloadV* as satisfying a fixture's expected consensus rejection, as long as the fixture carried a validationError. A payload refused with -38005 (unsupported fork) or -32602 (invalid params) is rejected before any consensus rule runs, so it cannot demonstrate the rejection the fixture asserts. EEST fixtures already say which payloads are meant to come back as an engine API error: each engineNewPayloads entry carries an optional errorCode. The harness ignored it. Bind it, and accept an RPC error only when the fixture asked for that exact code; assert the converse too, so a fixture expecting an errorCode fails when the payload is accepted for validation instead.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
BlockchainTestBase.AssertExpectedRpcErroraccepted any JSON-RPC error fromengine_newPayloadV*as satisfying a fixture's expected consensus rejection, as long as the fixture carried avalidationError:A payload refused with
-38005(unsupported fork) or-32602(invalid params) is rejected by the engine API before any consensus rule runs, so it cannot demonstrate the rejection an invalid-block fixture asserts. Every such payload scored green.EEST fixtures already say which payloads are meant to come back as an engine API error — each
engineNewPayloadsentry carries an optionalerrorCodealongsidevalidationError— and the harness ignored it entirely.errorCodeonTestEngineNewPayloadsJson, parsed byJsonToEthereumTest.ParseErrorCode(throws rather than silently returning null on a malformed value).DescribeUnexpectedRpcErroraccepts an RPC error only when the fixture asked for that exact code. NoerrorCode⇒ any error fails, with a message naming the code and saying the payload was never validated.errorCodenow fails if the payload was accepted for validation instead.EngineRpcErrorTestspins both directions pluserrorCodeparsing.Types of changes
What types of changes does your code introduce?
Testing
Requires testing
If yes, did you write tests?
Notes on testing
EngineRpcErrorTests(9 cases) covers the classifier in both directions anderrorCodeparsing.Blast radius was measured over the full
blockchain_tests_enginecorpus, not sampled.Corpus shape (
tests-glamsterdam-devnet@v8.1.0) — 10,971 files, 79,564 test cases, 98,376newPayloadcalls. Exactly 78 payloads carry anerrorCode, all-32602(osaka 20, prague 20, amsterdam 27, shanghai→cancun 6, bpo2→amsterdam 5). 10,313 payloads expect INVALID with avalidationError; the rest expect VALID.frames-devnet8-integration×tests-glamsterdam-devnet@v8.1.0(the pairing that branch's CI uses), full corpus, before and after:Identical, down to the same three pre-existing
test_invalid_layout_with_swapped_decodable_offsetsfailures. All 78 RPC errors observed were the 78 fixtures asking for-32602, answered with-32602. Zero unexpected protocol-level errors.master-base ×
tests-bal@v7.2.0(this branch's pinned archive), full corpus: 72,378 tests, 3 fail. Zero unexpected protocol-level errors here too, so the half this PR is named for costs nothing. Of the three:test_invalid_post_fork_block_without_bal_hash_field[fork_BPO2ToAmsterdamAtTime15k]andtest_invalid_pre_fork_block_with_bal_hash_field[fork_BPO2ToAmsterdamAtTime15k]— new, from the converse assertion. Both fixtures expect-32602; the client validates the payload and rejects it as an invalid block instead.ExecutionPayloadParams<T>.ValidateEngineApiVersionParamsdoes implement both directions of the BAL/newPayloadV5rule, so the check exists and simply did not fire for these two — most likely the spec resolved pre-fork for that payload. Needs a trace before attributing a cause. These two are the only reds; if they should not land, the converse assertion is the separable half.test_run_until_out_of_gas[fork_Cancun-...-tstore_wide_address_space]returning SYNCING — unrelated, a local 8-process contention artefact.Reproducing the false green. On master-base × v8.1.0,
test_invalid_pre_fork_block_with_bal_hash_field[fork_BPO2ToAmsterdamAtTime15k]passed while the client answered-32602 "Block access list must not be set before engine_newPayloadV5"at V4 — the fixture's consensus rule was never exercised. With this change:Documentation
Requires documentation update
Requires explanation in Release Notes
Remarks
The premise that a strict check would flip a large number of engine tests red does not survive measurement — it flips 0 on both matched corpora. The false green only fires when the client and the fixture archive are out of step, which is why it went unnoticed: the reported 5939/24805 and 59/135 figures came from a stale corpus and from EIP-8141 fixtures, not from what CI runs.
For branches paired with a foreign or stale corpus the new reds are the signal, and triage by code:
-38005means the spec provider does not recognise the fork carried by the payload (repin the archive, or wire the fork);-32602means a param-shape mismatch (endpoint version vs. fixture, or a field the endpoint does not accept yet). If such a branch must stay green while its pin catches up, skip the fixture tree at load time — a skipped test is honest, a green one is not.