Skip to content

fullhistory: add bench-ingest subcommand for ingestion benchmarks (#771)#857

Draft
marwen-abid wants to merge 1 commit into
feature/full-historyfrom
bench-ingest-771
Draft

fullhistory: add bench-ingest subcommand for ingestion benchmarks (#771)#857
marwen-abid wants to merge 1 commit into
feature/full-historyfrom
bench-ingest-771

Conversation

@marwen-abid

Copy link
Copy Markdown

What

Adds stellar-rpc bench-ingest cold|hot: benchmarks for the production
ingestion paths (ingest.WriteColdChunk, ingest.HotService). A CSV
MetricSink aggregates the signals the engine already emits into
per-stage percentile reports (one CSV per cold data type, hot.csv,
driver.csv) — no re-instrumentation, the engine under measurement is
the one that ships.

  • Sources: frozen local packs (--source=pack) or the backfill BSB
    backend (--source=bsb) against GCS or S3; cold runs N chunks with
    --chunk-workers fan-out, hot caps runs with --num-ledgers.
  • Guards: takes the daemon's exclusive flocks on every write root,
    refuses in-place repacks and pre-existing hot DBs, bounds --chunk
    at the last valid chunk ID. Failed/interrupted runs persist PARTIAL
    CSVs for completed work.
  • Supporting changes: ingest exports DataType*/Stage* label
    constants, fhtest.EventLCMBytes is the canonical event-bearing LCM
    fixture, geometry.LedgerPackPath owns the pack-path formula.

Example run (real pubnet data from the public S3 lake):

    stellar-rpc bench-ingest cold --source=bsb --datastore-type=S3 \
        --region=us-east-2 \
        --bucket-path=aws-public-blockchain/v1.1/stellar/ledgers/pubnet \
        --chunk=6100 --num-chunks=1 --types=events \
        --cold-out-dir=/tmp/cold --out=/tmp/bench-out

Smoke-tested on pubnet chunk 6100: cold events = 6.15M events, 14.2s of
ingest inside 46.8s wall (rest is S3 download); hot commit fsync
dominates at p50 7ms/ledger, as designed.

Note for CSV consumers: read_blocked's n is ledgers − 1 — the first
pull (source setup) is excluded so max/total reflect steady-state waits.

Why

We need baseline ingestion numbers for the full-history storage engine —
per-stage cold throughput and hot per-ledger latency — to tune it
(#836), to run the apply-load measurement campaign, and to seed the
bench-in-CI work (#775). Benchmarking the shipped entry points from a
subcommand keeps the numbers honest: what's measured is exactly what
runs in production.

Query benchmarks are split out to #856 (closes #771).

Add `stellar-rpc bench-ingest cold|hot`, benchmarking the production
ingest entry points (ingest.WriteColdChunk, ingest.HotService) as they
ship — no re-instrumentation. A CSV MetricSink aggregates the engine's
existing signals into per-stage percentile reports: one CSV per cold
data type, hot.csv, and driver.csv.

- Sources: frozen local packs (--source=pack) or the backfill BSB
  backend (--source=bsb) against GCS or S3; cold runs N chunks with
  --chunk-workers fan-out, hot caps runs with --num-ledgers.
- Guards: takes the daemon's exclusive flocks on every write root,
  refuses in-place repacks and pre-existing hot DBs, bounds --chunk at
  the last valid chunk ID.
- Failed or interrupted runs persist PARTIAL CSVs for completed work;
  --out is validated up front.
- Supporting: ingest exports DataType*/Stage* label constants (single
  vocabulary source for sinks), fhtest gains the canonical event-bearing
  LCM fixture, geometry.LedgerPackPath owns the pack-path formula.

Query benchmarks are tracked separately in #856.
@marwen-abid marwen-abid self-assigned this Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant