perf(l1): use 4KB RocksDB blocks for trie-node and flat-KV CFs#6940
perf(l1): use 4KB RocksDB blocks for trie-node and flat-KV CFs#6940edg-l wants to merge 5 commits into
Conversation
ACCOUNT_TRIE_NODES/STORAGE_TRIE_NODES and ACCOUNT_FLATKEYVALUE/ STORAGE_FLATKEYVALUE are pure exact-key point lookups on the execution read path. Drop their block size from 16KB (scan-tuned) to a page-sized 4KB to cut per-get read+search amplification. Write-time option: applies to newly flushed/compacted SSTs; existing SSTs are read as-is.
github-actions
Bot
added
L1
🤖 Kimi Code ReviewThis is a solid performance optimization with correct rationale. Summary
Minor Suggestions
No blocking issues. This follows established RocksDB tuning practices for LSM-tree workloads heavy on point lookups (OLTP-style vs. analytics). Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code ReviewNo findings. This PR only retunes RocksDB block size for Residual risk / testing gap: because this only affects newly written or compacted SSTs, the performance effect will be delayed and mixed on existing databases. I’d want benchmark evidence on a fresh sync or after forced compaction, plus block-cache hit/miss stats, before concluding the 4KB setting is a net win for these CFs. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
Greptile SummaryThis PR tunes RocksDB block sizing for hot execution state reads. The main changes are:
Confidence Score: 4/5The changed flow looks mergeable after checking the cache impact of the smaller blocks.
crates/storage/backend/rocksdb.rs
|
| Filename | Overview |
|---|---|
| crates/storage/backend/rocksdb.rs | Changes RocksDB block-based table options for trie-node and flat-KV column families from 16KB to 4KB blocks. |
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
crates/storage/backend/rocksdb.rs:195
**Shared Cache Metadata Pressure**
With `cache_index_and_filter_blocks(true)`, the smaller 4KB blocks create about four times as many index entries for the large trie-node CFs, and those index blocks compete in the same shared block cache as data and filters for every CF. On large state DBs or smaller cache configurations, newly flushed or compacted SSTs can evict hot data blocks and slow block import or RPC reads even though the point lookup itself reads less data.
### Issue 2 of 2
crates/storage/backend/rocksdb.rs:210
**Flat KV Index Expansion**
`STORAGE_FLATKEYVALUE` is likely one of the largest state CFs, and 4KB blocks substantially increase its per-SST index footprint while sharing the same cache with other column families. After compaction rewrites this CF, cache space that previously held data blocks can be consumed by extra index blocks, causing slower reads outside the flat-KV path as well.
Reviews (1): Last reviewed commit: "perf(l1): use 4KB RocksDB blocks for tri..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| block_opts.set_block_size(16 * 1024); // 16KB | ||
| // 4KB blocks: these CFs are pure exact-key point lookups | ||
| // (trie nodes / flat KV) on the execution read path, so a | ||
| // page-sized block minimizes per-get read+search amplification |
There was a problem hiding this comment.
With cache_index_and_filter_blocks(true), the smaller 4KB blocks create about four times as many index entries for the large trie-node CFs, and those index blocks compete in the same shared block cache as data and filters for every CF. On large state DBs or smaller cache configurations, newly flushed or compacted SSTs can evict hot data blocks and slow block import or RPC reads even though the point lookup itself reads less data.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/storage/backend/rocksdb.rs
Line: 195
Comment:
**Shared Cache Metadata Pressure**
With `cache_index_and_filter_blocks(true)`, the smaller 4KB blocks create about four times as many index entries for the large trie-node CFs, and those index blocks compete in the same shared block cache as data and filters for every CF. On large state DBs or smaller cache configurations, newly flushed or compacted SSTs can evict hot data blocks and slow block import or RPC reads even though the point lookup itself reads less data.
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| @@ -203,7 +208,9 @@ impl RocksDBBackend { | |||
| cf_opts.set_memtable_prefix_bloom_ratio(0.2); // Bloom filter | |||
|
|
|||
| let mut block_opts = BlockBasedOptions::default(); | |||
There was a problem hiding this comment.
STORAGE_FLATKEYVALUE is likely one of the largest state CFs, and 4KB blocks substantially increase its per-SST index footprint while sharing the same cache with other column families. After compaction rewrites this CF, cache space that previously held data blocks can be consumed by extra index blocks, causing slower reads outside the flat-KV path as well.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/storage/backend/rocksdb.rs
Line: 210
Comment:
**Flat KV Index Expansion**
`STORAGE_FLATKEYVALUE` is likely one of the largest state CFs, and 4KB blocks substantially increase its per-SST index footprint while sharing the same cache with other column families. After compaction rewrites this CF, cache space that previously held data blocks can be consumed by extra index blocks, causing slower reads outside the flat-KV path as well.
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
🤖 Claude Code ReviewNow I have verified findings across all angles. Let me compose the final review. PR Review:
|
# Conflicts: # CHANGELOG.md
The cache-floor sweep cited in the block-cache-size docs was measured at 16KB blocks for the trie-node/flat-KV CFs. The 4KB change (#6940) raises their index/filter block count ~4x, so the ~8GiB thrash floor is likely now somewhat higher and warrants a re-sweep.
|
Lines of code reportTotal lines added: Detailed view |
Benchmark Block Execution Results Comparison Against Main
|
| // vs the prior 16KB (scan-tuned). Write-time option: applies to | ||
| // newly flushed/compacted SSTs; existing SSTs are read as-is. | ||
| block_opts.set_block_size(4 * 1024); // 4KB | ||
| block_opts.set_bloom_filter(10.0, false); // 10 bits per key |
There was a problem hiding this comment.
Non-blocking follow-up thought: given the ~4x index/filter block-count growth on these two large CFs (already flagged in the block-cache-size doc), what do you think about pairing this later with partitioned filters (set_bloom_filter(10.0, true) + a two-level partitioned index)? That's the standard mitigation for exactly this pattern — small data block + large SST → filter/index memory blowup — because filter loads become finer-grained under cache eviction instead of the whole per-SST filter block being pulled in. The doc update on DEFAULT_ROCKSDB_BLOCK_CACHE_SIZE_BYTES covers this from the sizing side; partitioning would attack the same pressure from the CF-config side. Not for this PR, just checking if you'd already considered it.
4 participants
What
Lower the RocksDB data-block size from 16KB to 4KB on the four execution-read-path column families:
ACCOUNT_TRIE_NODES/STORAGE_TRIE_NODESACCOUNT_FLATKEYVALUE/STORAGE_FLATKEYVALUEWhy
These CFs are pure exact-key point lookups (trie nodes and flat KV) hit on the hot execution read path; there are no range scans. With a 16KB block, every cold random get pulls a full 16KB block off disk to return a ~32-byte value — ~4x read amplification. A page-sized 4KB block minimizes per-get read + in-block search amplification.
set_block_sizeis a write-time option: it applies to newly flushed/compacted SSTs. Existing SSTs are read with whatever block size they were written at, so the benefit accrues as the DB rewrites (or after a regen/reblock of a snapshot).Measurement
A/B on a cold, RAM-exceeding bloated state DB (
sstore_bloated10GB-class prestate, 300M-gas block), reblocking a copy of the same DB from 16KB to 4KB:The win is read-amplification driven, so it shows up most on cold, larger-than-RAM state where reads actually hit the device.
Scope
Bloom filters (
set_bloom_filter(10.0, false)) andmemtable_prefix_bloom_ratiowere already present on these CFs; this PR only changes the block size. No behavior change to other CFs (headers/bodies, codes, receipts, tx locations stay at their current sizes).