feat(bench): SSH-based distributed benchmark orchestration - #660
Merged
Conversation
Issue #98: lays the foundation for SSH-based distributed benchmark orchestration. `config.rs` parses/validates `nodes.yaml` (coordinator settings, per-node SSH destination/arch/features, tilde-expanded ssh_key). `ssh.rs` defines the `RemoteExec` trait with a `SystemSsh` implementation that shells to real `ssh`/`scp`, special-casing localhost to run commands directly with no wrapper — this keeps the local-node path real (not mocked) for later CI-safe integration testing. Adds `serde_yaml` as an optional dependency gated into the `bench-runner` feature.
Adds `Isolation` (Exclusive/Concurrent) to the benchmark registry, defaulting every entry to Exclusive since criterion/CLI timing benchmarks are skewed by a shared CPU; `nodes.yaml`'s `benchmarks[]` list overrides specific names to Concurrent on beefy nodes. `scheduler.rs` turns a node + benchmark list + overrides into an ordered list of "waves" (batches that run concurrently, bounded by `max_concurrent`) — pure data transformation, no threads or I/O, so the packing logic is fully unit-testable independent of execution.
`sync.rs` checks a node's reported `--version` against the local build, and if it differs (or `--force`), cross-compiles for the node's target triple and uploads the result. Cross-compilation is a local side effect (unlike everything else in `orchestrate/`, which talks to a node via `RemoteExec`), so it gets its own small `BuildRunner` abstraction — same dependency-injection shape, so tests never shell out to a real `cargo build`. `target_triple_for` lets `nodes.yaml` disambiguate cases `arch` alone can't (aarch64-apple-darwin vs aarch64-unknown-linux-gnu are both "aarch64").
`nodes.rs` reports each configured node's reachability — for EC2-backed nodes, its instance state (never SSH-probing a known-stopped instance); otherwise a plain SSH connectivity check, or "ready" immediately for localhost — and can start/stop those EC2 instances. AWS CLI invocation is a third distinct local side effect (alongside `RemoteExec` and `BuildRunner`), so it gets its own thin `Ec2Control` abstraction. `bench orchestrate` deliberately never auto-starts a stopped instance itself — that's a silent, cost-incurring side effect a user should choose explicitly via `bench nodes --start`.
`aggregate.rs` concatenates every node's per-benchmark `*.jsonl` result files into one `results.jsonl` plus a run-level `metadata.json` — recursively, since each `bench run <name>` invocation writes its output one (or more) directory levels under the node dir, not flat. `report.rs` is a separate, standalone `bench report` command: purely offline (no nodes.yaml, no SSH), it reads the `summary.json` each `bench run` invocation already writes plus each node's `node_info.json` (for `arch`), and renders a markdown report — a per-benchmark node-by-node comparison table, a per-architecture average-duration table, and, given `--baseline <prior-run>`, a regression table for anything more than `--threshold` slower than the same (node, benchmark) pair in that baseline. `--check` verifies the report file is current instead of rewriting it, for CI drift detection.
`executor.rs` is the `bench orchestrate` entry point tying the rest of the module together: each selected node runs on its own thread, waves run sequentially within a node (via scheduler.rs) with jobs inside a wave running concurrently, Ctrl+C is checked between waves for cooperative early stop, and a Node Prep step calls sync.rs's check-and-upload before executing unless --no-sync. Wires all four new subcommands (orchestrate/sync/nodes/report) into `BenchRunnerSubcommand` and `main.rs`'s dispatch. Adds `test_support.rs` (`FakeExec`/`FakeBuild`/`FakeAwsCli`), the only test-only code in the module, so scheduler/executor/sync/nodes logic is fully unit-testable without real SSH, cargo build, or aws CLI access. Adds `nodes.yaml.example` (the real file is gitignored — it encodes machine-specific hostnames/keys) and `.gitignore` entries for it and the distributed-run results directory.
Spawns the built binary directly (the locate_cli_tests.rs pattern): config validation (missing file, invalid YAML, unknown node selection), --dry-run side-effect-freedom, and a real localhost run exercising SystemSsh's is_local() bypass end-to-end (connectivity check, benchmark exec, download, aggregate, metadata) with no mocking and no network access. Real multi-node runs over Tailscale SSH are manual-only; see the new docs/guides/benchmarking.md section for that runbook.
Adds a "Distributed Benchmark Orchestration" section to docs/guides/benchmarking.md — setup (copy nodes.yaml.example), and one subsection each for bench nodes/sync/orchestrate/report — positioned right after the existing Unified Benchmark Runner section this automates the manual multi-machine workflow the guide otherwise describes by hand. Adds the equivalent quick-reference commands to CLAUDE.md's Common Commands.
`bench sync`'s cross-compile step built the binary it deploys to remote nodes with `--features cli`, but `bench orchestrate` invokes `bench run <name>` on that binary over SSH — and the entire `bench` subcommand tree is gated behind `--features bench-runner`, which is a strict superset of `cli`. A `cli`-only deploy passes `bench sync`'s own `--version` check (unaffected by the feature gate) but fails every real benchmark run with an unrecognized subcommand. Found while validating `bench sync`/`bench orchestrate` end-to-end against a live EC2 node over Tailscale — exactly the gap the existing fake-backed unit tests can't catch, since they never invoke a real cross-compiled binary.
Two additions from validating bench sync/bench orchestrate against a real EC2 node over SSH end-to-end (not just fakes): - testing skill: fakes standing in for "build an artifact, then invoke it across a process/host boundary" can't catch a mismatch between what's built and what's needed to run it — only a live run can. This exact pattern is what caught the bench-runner/cli feature-flag bug (ac5c78d). - benchmarking guide: SSH access layers with per-connection ACLs enforce their own username policy independent of key auth, so an omitted username in `host` can silently fail even with a valid key. Also adds a bench sync workaround for targets without a local cross-compiler toolchain: build natively on the node and run with --no-sync.
newhoggy
force-pushed
the
issue-98-ssh-bench-orchestration
branch
from
August 7, 2026 15:26
641abef to
a38ab20
Compare
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nodes.yamlconfig parsing, isolation-aware wave scheduling, binary sync (cross-compile + deploy), EC2 lifecycle management, and cross-node result aggregation/reporting, wired up asbench orchestrate/bench sync/bench nodes/bench reportCLI subcommandsdocs/guides/benchmarking.mdbench synccross-compiled deployed binaries with--features cliinstead of--features bench-runner, so a synced binary would be missing thebenchsubcommandbench orchestrateneeds to invoke on itTest plan
cargo test— all 49orchestrate::unit tests passcargo build --release --features bench-runnertests/orchestrate_cli_tests.rs)bench orchestrateagainst a real EC2 Graviton4 node over SSH —rank_select(325.7s) andpopcount_strategies(945.4s) both passed, results/node_info.jsonaggregated correctlybench sync's cross-compile step itself not yet exercised live (validated by building natively on the remote node instead, since no local aarch64-Linux cross-linker was available).jsonl/bench reportcomparison path not yet exercised live (only run against Criterion-type benchmarks, which report pass/fail + duration only —.jsonlis produced by CLI-type benchmarks likejq_bench/yq_bench)Addresses #98.