Skip to content

Add EthCallChaos benchmark/compare/isolated configs + per-category scenarios #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cbermudez97 merged 13 commits into from
Jul 28, 2026
Merged

Add EthCallChaos benchmark/compare/isolated configs + per-category scenarios #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
da24412
bench: add ethcallchaos-load benchmark config (force-add past gitignore)
kamilchodola Jul 24, 2026
5342108
bench: add HEAVY ethcallchaos workload (30-500ms calls) for a real lo…
kamilchodola Jul 24, 2026
8622bfe
bench: v2 heavy workload (multi-second Multicall3 calls from robust-f…
kamilchodola Jul 26, 2026
4b4e88a
compare: heavy-v2 cross-client equality config (validate responses, n…
kamilchodola Jul 27, 2026
bf18a27
@
kamilchodola Jul 27, 2026
dc15a18
@
kamilchodola Jul 27, 2026
3ef958c
per-category validated regression set (15 scenarios, ranked by valida…
kamilchodola Jul 28, 2026
908f580
per-category cross-client compare config (same-head parity + latency)
kamilchodola Jul 28, 2026
bbb38a6
fix compare-config ids to [a-zA-Z0-9_-]
kamilchodola Jul 28, 2026
7804c83
isolated single/low-gas#1 config for NM-vs-reth contention test
kamilchodola Jul 28, 2026
61a3e66
bump vus=512 on isolated single-low-gas-1 (avoid VU cap at 1000rps)
kamilchodola Jul 28, 2026
b0a0112
per-scenario isolated configs for isolated+mixed sweep
kamilchodola Jul 28, 2026
b7e35f1
Move ethcallchaos generator under rpc-calls/scripts, annotate SOURCES
cbermudez97 Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions config/benchmark/ethcallchaos-heavy2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
test_name: "EthCallChaos v2 HEAVY (multi-second)"
description: "Adversarial eth_call mix evolved by EthCallChaos (v2 heavy 150ms-4.4s multicall), replayed at a target rps to compare how each client handles load. clients/rps/vus are set by the runner."
clients:
- nethermind
duration: "30s"
rps: 100
vus: 100
calls:
- name: "ethcallchaos-heavy2"
file: "./rpc-calls/ethcallchaos-heavy2.jsonl"
file_type: "jsonl"
weight: 1
12 changes: 12 additions & 0 deletions config/benchmark/ethcallchaos-load-heavy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
test_name: "EthCallChaos HEAVY load"
description: "Adversarial eth_call mix evolved by EthCallChaos (heavy 30-500ms), replayed at a target rps to compare how each client handles load. clients/rps/vus are set by the runner."
clients:
- nethermind
duration: "30s"
rps: 100
vus: 100
calls:
- name: "ethcallchaos-heavy"
file: "./rpc-calls/ethcallchaos-heavy.jsonl"
file_type: "jsonl"
weight: 1
12 changes: 12 additions & 0 deletions config/benchmark/ethcallchaos-load.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
test_name: "EthCallChaos corpus load"
description: "Adversarial eth_call mix evolved by EthCallChaos (bounded <300ms), replayed at a target rps to compare how each client handles load. clients/rps/vus are set by the runner."
clients:
- nethermind
duration: "30s"
rps: 100
vus: 100
calls:
- name: "ethcallchaos-corpus"
file: "./rpc-calls/ethcallchaos-corpus.jsonl"
file_type: "jsonl"
weight: 1
172 changes: 172 additions & 0 deletions config/benchmark/ethcallchaos-percategory-validated.yaml
View file
Open in desktop

Large diffs are not rendered by default.

337 changes: 337 additions & 0 deletions config/benchmark/ethcallchaos-percategory.yaml
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 18 additions & 0 deletions config/benchmark/iso/multicall-high-gas-1.yaml
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 18 additions & 0 deletions config/benchmark/iso/multicall-high-gas-2.yaml
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 18 additions & 0 deletions config/benchmark/iso/multicall-high-gas-3.yaml
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 18 additions & 0 deletions config/benchmark/iso/multicall-large-1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
test_name: "multicall/large#1 isolated"
clients:
- nethermind
duration: "60s"
rps: 150
vus: 512
calls:
- name: "multicall/large#1"
method: "eth_call"
params:
- to: "0xcA11bde05977b3631167028862bE2a173976CA11"
data: "0x82ad56cb00000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000028000000000000000000000000000000000000000000000000000000000000003c0000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000005a00000000000000000000000000000000000000000000000000000000000000640000000000000000000000000000000000000000000000000000000000000074000000000000000000000000000000000000000000000000000000000000007e0000000000000000000000000000000000000000000000000000000000000092000000000000000000000000000000000000000000000000000000000000009c00000000000000000000000000000000000000000000000000000000000000ac00000000000000000000000000000000000000000000000000000000000000b800000000000000000000000000000000000000000000000000000000000000c400000000000000000000000000000000000000000000000000000000000000d000000000000000000000000000000000000000000000000000000000000000e400000000000000000000000000000000000000000000000000000000000000f8000000000000000000000000000000000000000000000000000000000000010c000000000000000000000000000000000000000000000000000000000000011c000000000000000000000000000000000000000000000000000000000000012800000000000000000000000000000000000000000000000000000000000001340000000000000000000000000000000000000000000000000000000000000148000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a000000000000000000000000A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB480000000000000000000000006B175474E89094C44Da98b954EedeAC495271d0F04f9f06492662567765185f85c668e5c6550a0f9584e2b3579ad5fc28f0922e300000000000000000000000000000000000000000000000000000000000001f400000000000000000000000069bba45e32f23d750ca1a474173b89f560b0a7790000000000000000000000000000000000000000000000000000000000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a000000000000000000000000A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB480000000000000000000000006B175474E89094C44Da98b954EedeAC495271d0F04f9f06492662567765185f85c668e5c6550a0f9584e2b3579ad5fc28f0922e300000000000000000000000000000000000000000000000000000000000001f400000000000000000000000069bba45e32f23d750ca1a474173b89f560b0a77900000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C7000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004bb7b8b8000000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C7000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004bb7b8b8000000000000000000000000000000000000000000000000000000000000000000000000000000000D51a44d3FaE010294C616388b506AcdA1bfAAE46000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000064556d6e9f00000000000000000000000000000000000000000000000005db02e6168d7a4000000000000000000000000000000000000000000000000001c4b7734ab017c000000000000000000000000000000000000000000000000005b0419d2481398000000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C7000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004bb7b8b800000000000000000000000000000000000000000000000000000000000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a000000000000000000000000A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB480000000000000000000000006B175474E89094C44Da98b954EedeAC495271d0F04f9f06492662567765185f85c668e5c6550a0f9584e2b3579ad5fc28f0922e300000000000000000000000000000000000000000000000000000000000001f400000000000000000000000069bba45e32f23d750ca1a474173b89f560b0a779000000000000000000000000000000000000000000000000000000000000000000000000000000003d9819210A31b4961b30EF54bE2aeD79B9c9Cd3B000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004b0772d0b00000000000000000000000000000000000000000000000000000000000000000000000000000000DC24316b9AE028F1497c275EB9192a3Ea0f670220000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000645e0d443f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000805382b0d42ad000000000000000000000000000000000000000000000000000000000000000000000000000000000083F20F44975D03b1b09e64809B757c47f942BEeA00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002407a2d13a00000000000000000000000000000000000000000000000008d254921e262600000000000000000000000000000000000000000000000000000000000000000000000000000000003d9819210A31b4961b30EF54bE2aeD79B9c9Cd3B0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000245ec88c79000000000000000000000000ae2Fc483527B8EF99EB5D9B44875F005ba1FaE13000000000000000000000000000000000000000000000000000000000000000000000000000000003d9819210A31b4961b30EF54bE2aeD79B9c9Cd3B0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000245ec88c79000000000000000000000000DFd5293D8e347dFe59E90eFd55b2956a1343963d0000000000000000000000000000000000000000000000000000000000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a000000000000000000000000A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB480000000000000000000000006B175474E89094C44Da98b954EedeAC495271d0F04f9f06492662567765185f85c668e5c6550a0f9584e2b3579ad5fc28f0922e300000000000000000000000000000000000000000000000000000000000001f400000000000000000000000069bba45e32f23d750ca1a474173b89f560b0a7790000000000000000000000000000000000000000000000000000000000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a000000000000000000000000A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB480000000000000000000000006B175474E89094C44Da98b954EedeAC495271d0F04f9f06492662567765185f85c668e5c6550a0f9584e2b3579ad5fc28f0922e300000000000000000000000000000000000000000000000000000000000001f400000000000000000000000069bba45e32f23d750ca1a474173b89f560b0a7790000000000000000000000000000000000000000000000000000000000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a00000000000000000000000047ac0Fb4F2D84898e4D9E7b4DaB3C24507a6D50300000000000000000000000047ac0Fb4F2D84898e4D9E7b4DaB3C24507a6D5030000000000000000000000000000000000000000000000000bd6a280489b53800000000000000000000000000000000000000000000000000000000000002710000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C70000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000645e0d443f00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000025c6d849f2c71600000000000000000000000000000000000000000000000000000000000000000000000000000000087870Bca3F3fD6335C3F4ce8392D69350B4fA4E200000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002435ea6a75000000000000000000000000C02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2000000000000000000000000000000000000000000000000000000000000000000000000000000003d9819210A31b4961b30EF54bE2aeD79B9c9Cd3B0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000245ec88c7900000000000000000000000021a31Ee1afC51d94C2eFcCAa2092aD10282855490000000000000000000000000000000000000000000000000000000000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a00000000000000000000000047ac0Fb4F2D84898e4D9E7b4DaB3C24507a6D50300000000000000000000000047ac0Fb4F2D84898e4D9E7b4DaB3C24507a6D5030000000000000000000000000000000000000000000000000bd6a280489b538000000000000000000000000000000000000000000000000000000000000027100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003d9819210A31b4961b30EF54bE2aeD79B9c9Cd3B000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004b0772d0b00000000000000000000000000000000000000000000000000000000"
gas: "0x5f5e100"
value: "0x0"
from: "0xDA9dfA130Df4dE4673b89022EE50ff26f6EA73Cf"
- "latest"
weight: 1
thresholds: ["p(99)<600000"]
18 changes: 18 additions & 0 deletions config/benchmark/iso/multicall-med-1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
test_name: "multicall/med#1 isolated"
clients:
- nethermind
duration: "60s"
rps: 150
vus: 512
calls:
- name: "multicall/med#1"
method: "eth_call"
params:
- to: "0xcA11bde05977b3631167028862bE2a173976CA11"
data: "0x82ad56cb0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000001200000000000000000000000000000000000000000000000000000000000000260000000000000000000000000000000000000000000000000000000000000030000000000000000000000000000000000000000000000000000000000000003a000000000000000000000000000000000000000000000000000000000000004a0000000000000000000000000000000000000000000000000000000000000054000000000000000000000000000000000000000000000000000000000000005e000000000000000000000000000000000000000000000000000000000000006e000000000000000000000000000000000000000000000000000000000000007e000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a000000000000000000000000A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB480000000000000000000000006B175474E89094C44Da98b954EedeAC495271d0F04f9f06492662567765185f85c668e5c6550a0f9584e2b3579ad5fc28f0922e300000000000000000000000000000000000000000000000000000000000001f400000000000000000000000069bba45e32f23d750ca1a474173b89f560b0a77900000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C7000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004bb7b8b8000000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C7000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004bb7b8b8000000000000000000000000000000000000000000000000000000000000000000000000000000000D51a44d3FaE010294C616388b506AcdA1bfAAE46000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000064556d6e9f00000000000000000000000000000000000000000000000005db02e6168d7a4000000000000000000000000000000000000000000000000001c4b7734ab017c000000000000000000000000000000000000000000000000005b0419d2481398000000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C7000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004bb7b8b80000000000000000000000000000000000000000000000000000000000000000000000000000000003d9819210A31b4961b30EF54bE2aeD79B9c9Cd3B000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004b0772d0b00000000000000000000000000000000000000000000000000000000000000000000000000000000D51a44d3FaE010294C616388b506AcdA1bfAAE46000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000064556d6e9f00000000000000000000000000000000000000000000000005873324de2712400000000000000000000000000000000000000000000000000c73a0a29e6c3f000000000000000000000000000000000000000000000000000aa3f037797d460000000000000000000000000000000000000000000000000000000000000000000000000000000000DC24316b9AE028F1497c275EB9192a3Ea0f670220000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000645e0d443f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000805382b0d42ad000000000000000000000000000000000000000000000000000000000000000000000000000000000083F20F44975D03b1b09e64809B757c47f942BEeA00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002407a2d13a00000000000000000000000000000000000000000000000008d254921e26260000000000000000000000000000000000000000000000000000000000"
gas: "0x1c9c380"
value: "0x0"
from: "0xA7EFAe728D2936e78BDA97dc267687568dD593f3"
- "latest"
weight: 1
thresholds: ["p(99)<600000"]
18 changes: 18 additions & 0 deletions config/benchmark/iso/multicall-small-1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
test_name: "multicall/small#1 isolated"
clients:
- nethermind
duration: "60s"
rps: 150
vus: 512
calls:
- name: "multicall/small#1"
method: "eth_call"
params:
- to: "0xcA11bde05977b3631167028862bE2a173976CA11"
data: "0x82ad56cb000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000018000000000000000000000000061fFE014bA17989E743c5F6cB21bF9697530B21e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a4c6a5026a000000000000000000000000A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB480000000000000000000000006B175474E89094C44Da98b954EedeAC495271d0F04f9f06492662567765185f85c668e5c6550a0f9584e2b3579ad5fc28f0922e300000000000000000000000000000000000000000000000000000000000001f400000000000000000000000069bba45e32f23d750ca1a474173b89f560b0a77900000000000000000000000000000000000000000000000000000000000000000000000000000000bEbc44782C7dB0a1A60Cb6fe97d0b483032FF1C7000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000004bb7b8b8000000000000000000000000000000000000000000000000000000000"
gas: "0x1dcd6500"
value: "0x0"
from: "0xA7EFAe728D2936e78BDA97dc267687568dD593f3"
- "latest"
weight: 1
thresholds: ["p(99)<600000"]
Loading