Network/Scheduling Improvements - #5450
Merged
SirTyson merged 14 commits intoSep 11, 2026
Merged
Conversation
recvTransaction already submits the envelope to the Rust overlay. Remove the subsequent broadcast that submitted it a second time.
Run outbound I/O in owned tasks with bounded tx-set send admission. Reserve fetches before dispatch and give requests a separate stream from large responses, preserving ordering, shutdown, and retry-after-write-failure behavior. Cover blocked peers, request progress, framing, capacity, and cancellation with real-QUIC and unit tests.
Preview the first two nomination leader selections without mutating SCP, and schedule private construction after ledger close. Retain the validated close time and ledger identity until the existing trigger; refresh selection-unlimited snapshots and discard stale proposals on lifecycle transitions. Cover leader selection, timer gating, time bounds, classic and Soroban reuse, refill, and cancellation. Focused early-nomination tests, Herder/SCP regression checks, and overlay IPC tests pass.
Apply a fixed receive-buffer request to every listener and dial-only socket. Verify the effective OS size, use the Linux per-socket increase when needed, and reject sockets that cannot meet the requirement. Bind before starting App so a listener failure aborts initialization with its cause. Remove overlay and transport buffer settings and environment switches. Reject obsolete TOML fields, cover clamping and permission failures in unit tests, and document the mandatory OS allowance and reproducible vendor patch.
Cache network prefetches until Core requests their hash. Serialize demand and arrivals on the App task, satisfy pending requests once, preserve later explicit requests, and expire pending metadata with the existing cache window. Cover both event orderings, duplicates, eviction, local caching, and IPC enqueue failure with socket tests.
Retain one immutable allocation through cache lookup, send admission, and network writes. Write the frame prefix, discriminant, and payload under the existing stream lock without assembling another bulk copy. Preserve wire-byte admission accounting and IPC framing. Test App fanout ownership, queued response lifetimes, eviction, shutdown, and partial writes.
Enqueue network admission directly in the Core command FIFO before any awaited bookkeeping. Hold shared capacity through insertion, and mark flooding dedup only after successful enqueue. Remove the separate App TX queue while preserving bounded network admission and allowing genuinely later arrivals. Test ordering, saturation, retry, duplicate handling, cancellation, shutdown, and Core removal; update the subsystem docs.
SirTyson
force-pushed
the
mempool-reliability-20260903
branch
from
September 9, 2026 22:38
38f247f to
1a0e5fa
Compare
Wait for individual direct SCP writes to complete before acknowledging them to the state-response sender. Keep network I/O in dispatcher-owned tasks so blocked peers do not prevent other peers or shutdown from progressing. Add regression coverage for pending write acknowledgements, shutdown cancellation, and recent-slot-before-checkpoint delivery across real QUIC peers.
The generic generated-directory exclusion removed checked-in protobuf bindings required by libp2p-core, breaking the RPC image build. Keep this vendored source directory in the context.
Use a fixed unanimous quorum and exercise each member as the local node. Varying only remote keys could fail to select the fixed local node as the second leader for some CI seeds. Assert all members agree on the first two leaders and cover all three preparation roles.
SirTyson
force-pushed
the
mempool-reliability-20260903
branch
2 times, most recently
from
September 10, 2026 17:41
83ee3f7 to
893c27f
Compare
SirTyson
marked this pull request as ready for review
September 10, 2026 17:42
SirTyson
requested review from
bboston7 and
drebelsky
and
a balanced review from Copilot
September 10, 2026 17:42
Contributor
There was a problem hiding this comment.
Pull request overview
This PR targets throughput and tail-latency improvements in the networking/scheduling path (overlay v2 / QUIC), including reducing redundant TX/TX-set handling, adding early leader TX-set preparation hooks, and enabling QUIC stream prioritization via a vendored libp2p patch set.
Changes:
- Fix loadgen double-submission by removing redundant overlay broadcast after
Herder::recvTransaction. - Add a read-only SCP API to predict nomination leaders without mutating live slot state, with new unit tests.
- Vendor and patch libp2p crates to expose QUIC stream priority controls, plus supporting build/docs updates and Rust overlay TX-set caching/copy reductions.
Reviewed changes
Copilot reviewed 21 out of 149 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/simulation/LoadGenerator.cpp | Avoids double TX submission by removing redundant broadcast. |
| src/scp/test/SCPUnitTests.cpp | Adds tests ensuring leader preview has no SCP side effects and preserves tie/weight behavior. |
| src/scp/SCP.h | Adds predictNominationLeaders API declaration. |
| src/scp/SCP.cpp | Implements read-only nomination leader preview via a temporary Slot. |
| src/scp/NominationProtocol.h | Declares NominationProtocol::predictLeaders helper. |
| src/scp/NominationProtocol.cpp | Implements leader prediction by replaying leader-round selection without nomination side effects. |
| src/overlay/test/OverlayIPCTests.cpp | Removes pre-Soroban TX-set handling test; retains Soroban TX-set handling test. |
| src/Makefile.am | Ensures Rust overlay rebuilds when vendored crates / workspace lockfiles change. |
| src/herder/HerderImpl.h | Adds prepared-TX-set state/timer hooks and a deterministic testing injection point. |
| overlay/vendor/UPSTREAM.json | Records upstream crate SHAs/checksums for vendored libp2p components. |
| overlay/vendor/multistream-select/LICENSE | Adds vendored crate license. |
| overlay/vendor/multistream-select/CHANGELOG.md | Adds vendored crate changelog. |
| overlay/vendor/multistream-select/Cargo.toml.orig | Adds original manifest for vendored crate. |
| overlay/vendor/multistream-select/Cargo.toml | Adds Cargo-normalized manifest for vendored crate. |
| overlay/vendor/multistream-select/.cargo_vcs_info.json | Adds VCS info for vendored crate. |
| overlay/vendor/libp2p-swarm/tests/ui/fail/to_swarm_not_string.stderr | Adds vendored UI test artifact. |
| overlay/vendor/libp2p-swarm/tests/ui/fail/to_swarm_not_string.rs | Adds vendored UI test source. |
| overlay/vendor/libp2p-swarm/tests/ui/fail/prelude_not_string.stderr | Adds vendored UI test artifact. |
| overlay/vendor/libp2p-swarm/tests/ui/fail/prelude_not_string.rs | Adds vendored UI test source. |
| overlay/vendor/libp2p-swarm/tests/listener.rs | Adds vendored behaviour listener test. |
| overlay/vendor/libp2p-swarm/tests/connection_close.rs | Adds vendored connection-close behaviour test. |
| overlay/vendor/libp2p-swarm/src/translation.rs | Adds vendored translation utility module. |
| overlay/vendor/libp2p-swarm/src/stream.rs | Adds stream wrapper exposing transport stream priority APIs. |
| overlay/vendor/libp2p-swarm/src/stream_protocol.rs | Adds vendored stream protocol abstraction. |
| overlay/vendor/libp2p-swarm/src/listen_opts.rs | Adds vendored listen options type. |
| overlay/vendor/libp2p-swarm/src/handler/pending.rs | Adds vendored pending handler implementation. |
| overlay/vendor/libp2p-swarm/src/handler/map_out.rs | Adds vendored handler output mapping utility. |
| overlay/vendor/libp2p-swarm/src/handler/map_in.rs | Adds vendored handler input mapping utility. |
| overlay/vendor/libp2p-swarm/src/executor.rs | Adds vendored executor abstraction. |
| overlay/vendor/libp2p-swarm/src/dummy.rs | Adds vendored dummy behaviour/handler. |
| overlay/vendor/libp2p-swarm/src/connection/supported_protocols.rs | Adds vendored supported-protocol tracking. |
| overlay/vendor/libp2p-swarm/src/connection/pool/concurrent_dial.rs | Adds vendored concurrent dial implementation. |
| overlay/vendor/libp2p-swarm/src/connection/error.rs | Adds vendored connection error types. |
| overlay/vendor/libp2p-swarm/src/behaviour/listen_addresses.rs | Adds vendored listen-address tracking helper. |
| overlay/vendor/libp2p-swarm/src/behaviour/external_addresses.rs | Adds vendored external-address tracking helper. |
| overlay/vendor/libp2p-swarm/src/behaviour/either.rs | Adds vendored Either behaviour adapter. |
| overlay/vendor/libp2p-swarm/LICENSE | Adds vendored crate license. |
| overlay/vendor/libp2p-swarm/Cargo.toml.orig | Adds original manifest for vendored crate. |
| overlay/vendor/libp2p-swarm/Cargo.toml | Adds Cargo-normalized manifest for vendored crate. |
| overlay/vendor/libp2p-swarm/.cargo_vcs_info.json | Adds VCS info for vendored crate. |
| overlay/vendor/libp2p-quic/tests/stream_compliance.rs | Adds vendored QUIC stream compliance test. |
| overlay/vendor/libp2p-quic/src/provider/tokio.rs | Adds vendored tokio provider implementation. |
| overlay/vendor/libp2p-quic/src/provider/async_std.rs | Adds vendored async-std provider implementation. |
| overlay/vendor/libp2p-quic/src/provider.rs | Adds vendored provider trait/runtime enum. |
| overlay/vendor/libp2p-quic/src/lib.rs | Adds vendored QUIC transport crate (includes stream priority surface). |
| overlay/vendor/libp2p-quic/src/hole_punching.rs | Adds vendored hole-punching implementation. |
| overlay/vendor/libp2p-quic/src/connection/stream.rs | Adds vendored QUIC stream wrapper (priority + IO). |
| overlay/vendor/libp2p-quic/src/connection/connecting.rs | Adds vendored QUIC connecting future. |
| overlay/vendor/libp2p-quic/src/connection.rs | Adds vendored QUIC muxer implementation (hooks for priority). |
| overlay/vendor/libp2p-quic/LICENSE | Adds vendored crate license. |
| overlay/vendor/libp2p-quic/CHANGELOG.md | Adds vendored crate changelog. |
| overlay/vendor/libp2p-quic/Cargo.toml.orig | Adds original manifest for vendored crate. |
| overlay/vendor/libp2p-quic/Cargo.toml | Adds Cargo-normalized manifest for vendored crate. |
| overlay/vendor/libp2p-quic/.cargo_vcs_info.json | Adds VCS info for vendored crate. |
| overlay/vendor/libp2p-core/tests/transport_upgrade.rs | Adds vendored transport upgrade test. |
| overlay/vendor/libp2p-core/src/upgrade/select.rs | Adds vendored upgrade selection implementation. |
| overlay/vendor/libp2p-core/src/upgrade/ready.rs | Adds vendored ready-upgrade implementation. |
| overlay/vendor/libp2p-core/src/upgrade/pending.rs | Adds vendored pending-upgrade implementation. |
| overlay/vendor/libp2p-core/src/upgrade/error.rs | Adds vendored upgrade error types. |
| overlay/vendor/libp2p-core/src/upgrade/either.rs | Adds vendored either-upgrade adapter. |
| overlay/vendor/libp2p-core/src/upgrade/denied.rs | Adds vendored denied-upgrade implementation. |
| overlay/vendor/libp2p-core/src/transport/optional.rs | Adds vendored optional transport wrapper. |
| overlay/vendor/libp2p-core/src/transport/map.rs | Adds vendored map transport wrapper. |
| overlay/vendor/libp2p-core/src/transport/map_err.rs | Adds vendored map_err transport wrapper. |
| overlay/vendor/libp2p-core/src/transport/dummy.rs | Adds vendored dummy transport. |
| overlay/vendor/libp2p-core/src/transport/choice.rs | Adds vendored transport choice (OrTransport). |
| overlay/vendor/libp2p-core/src/lib.rs | Adds vendored libp2p-core crate root module. |
| overlay/vendor/libp2p-core/src/generated/peer_record.proto | Adds vendored generated protobuf source. |
| overlay/vendor/libp2p-core/src/generated/peer_record_proto.rs | Adds vendored generated protobuf bindings. |
| overlay/vendor/libp2p-core/src/generated/mod.rs | Adds vendored generated module index. |
| overlay/vendor/libp2p-core/src/generated/envelope.proto | Adds vendored generated protobuf source. |
| overlay/vendor/libp2p-core/src/generated/envelope_proto.rs | Adds vendored generated protobuf bindings. |
| overlay/vendor/libp2p-core/LICENSE | Adds vendored crate license. |
| overlay/vendor/libp2p-core/Cargo.toml.orig | Adds original manifest for vendored crate. |
| overlay/vendor/libp2p-core/Cargo.toml | Adds Cargo-normalized manifest for vendored crate. |
| overlay/vendor/libp2p-core/.cargo_vcs_info.json | Adds VCS info for vendored crate. |
| overlay/vendor/.gitignore | Ensures generated protobuf bindings aren’t ignored in vendored copy. |
| overlay/vendor/.gitattributes | Preserves whitespace in vendored files/patch context. |
| overlay/src/xdr.rs | Restricts TX-set framing helper to tests only. |
| overlay/src/libp2p_overlay/framed_io.rs | Adds segmented frame writer to reduce bulk allocations; includes tests. |
| overlay/src/ipc/transport.rs | Avoids TX-set payload clone by taking &[u8] instead of Vec<u8>. |
| overlay/src/flood/txset.rs | Shares cached TX-set XDR via Arc<Vec<u8>> to reduce copies. |
| overlay/src/flood/mempool.rs | Updates mempool insert doc comment to match new duplicate/capacity behavior. |
| overlay/src/config.rs | Makes overlay config parsing strict via deny_unknown_fields; adds test. |
| Makefile.am | Ensures top-level build tracks vendored overlay inputs and includes vendor in dist. |
| docs/rust-overlay/tx-propagation.md | Updates TX flooding/backpressure and stream separation documentation. |
| docs/rust-overlay/peer-connections.md | Updates peer-limit documentation to reflect actual overlay behavior. |
| docs/rust-overlay/mempool.md | Documents direct network admission path and ordering implications. |
| docs/rust-overlay/ipc.md | Updates IPC schema (TX-set request/cache includes slot) and channel discipline docs. |
| Cargo.toml | Adds [patch.crates-io] to use vendored libp2p crates and excludes vendor from workspace. |
| Cargo.lock | Updates lock to reflect path-patched vendored libp2p crates (no registry checksums). |
| .github/workflows/rpc.yml | Removes RPC integration test workflow. |
| .dockerignore | Ensures vendored generated protobuf bindings aren’t excluded from Docker context. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Remove the pre-Soroban-only fixture and the RPC integration CI workflow. Retain generalized tx-set tests and the existing Core CI jobs. Make the real-time "SCP checkpoint" out-of-sync sections robust on loaded CI runners: slow ledgers to 2s so the checkpoint slot is not superseded before the joining nodes process the SCP state, start the joining nodes only once the checkpoint slot is detached from the retained window, and latch the observed buffered-checkpoint and catchup-started state instead of requiring both to hold at one sampled instant within a 4s budget derived from the unused SEND_LATEST_CHECKPOINT_DELAY. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SirTyson
force-pushed
the
mempool-reliability-20260903
branch
from
September 10, 2026 19:21
893c27f to
012e62f
Compare
| REQUIRE(simulation->haveAllExternalized(5, 2)); | ||
| LOG_INFO(DEFAULT_LOG, "✓ Pre-Soroban consensus works with Rust overlay"); | ||
| } | ||
|
|
||
| /** |
A node that is out of sync learns the network state from peers' SCP-state responses, which carry the most recent slots plus the most recent checkpoint slot, with a gap between them that nothing will ever fill. Each slot is handed to SCP once its tx set is available, so when the checkpoint slot's tx set arrives first the node externalizes that slot and starts tracking it. Tracking mode then only consumes envelopes up to the next slot, which never comes, and the node sat idle until the consensus-stuck timer fired before it looked at the recent slots. When an out-of-sync node is tracking a slot for which no successor has ever been seen while later slots are ready, stop tracking, hand the smallest later slot to SCP, and drain the ready slots that follow. The lost-sync step matters because the driver rejects values for slots beyond the next tracked one while tracking. In-sync nodes are unaffected. Add a herder unit test that feeds a watcher externalize envelopes for a checkpoint slot and four later slots in both orders and checks it ends up tracking the latest slot with all five buffered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 151 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/rpc.yml:1
- This deletes the only RPC integration workflow, including its pull-request, push, and nightly coverage, without a replacement. That is unrelated to the networking/scheduling scope described for this PR and silently removes an entire CI signal; please restore the workflow (or move its replacement into this PR explicitly).
3 tasks
bboston7
approved these changes
Sep 11, 2026
bboston7
left a comment
bboston7
left a comment
Contributor
There was a problem hiding this comment.
As we discussed, I did a heavily AI assisted review of this huge change. I also looked at tests. The PR description seems accurate, and nothing jumped out to me as immediately incorrect. Given that this is going into an experimental prototype that will not directly feed into master, I think that level of review is sufficient to merge this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR contains a collection of implementation specific fixes and optimizations. There aren't really any design or logical changes (with the exception of early tx set preparation), but just a bunch of cleanups and optimizations at the scheduling and networking stack layer. I think these are all cheap, uncontroversial wins that remove some bottlenecks that might affect future larger experiments.
Each change has it's own commit:
f3dcf1357Fixes a loadgen bug, were we'd always submit a TX to our mempool twice.c87b8750bEnables concurrent TX set emission. Previously, while the underlying transport layer was parallel, the command dispatcher blocked on completion of some workloads, like tx sets. This means the leader had a de-facto serial scheduling conflict. This commit removes a bunch of unnecessary blocking at the command dispatch level. It also has a few other scheduling optimizations, like separating fetch request from the actual response write in the scheduler.f85d70defPrepare tx sets in advance. This is the only logical change. TX set preparation takes a long time for the leader. However, if we finish a ledger early, the leader just waits idle until the next ledger trigger, then starts construction. This starts construction early, such that the leader node starts preparing a tx set immediately after closing the last ledger. We also extend this to the 2nd eligible leader, where they also prepare a tx set in advance so we don't stall further on nomination time out.368fa9044Stream priorities. This implements stream priorities, such the SCP messages are highest priority, TX Sets next, then transaction flooding at lowest priority. This is all done via QUIC native capabilities. We also move some of the message types previously in the TX Set stream into the SCP stream. Specifically, the tx set fetching (and in the future, DONT_HAVE messages) all get assigned in the high priority SCP stream, where the actual tx set payload comes through the TX set stream. Basically, any small message related to consensus (SCP messages and "control" messages related to tx set fetching) are highest priority in SCP stream.4db17d26fUDP buffer settings. This changes a few low level UDP settings. During load, we actually have a lot of dropped packets, so I (claude) tinkered with the network settings. This made a surprisingly large difference.842e4e690Only deliver TX sets to core on request. This change makes is such that the rust overlay subprocess will cache incoming tx sets and only give them to core when requested. TX set serialization across C++ and rust is expensive, and the issue is that rust would sent multiple copies of the same TX set to C++ before C++ told rust that it had got and processed the tx set. This removes this duplication so we process a given TX set just once.ff620f2feReduce tx set copies. We maintain less redundant copies and allocations when a leader is transmitting TX sets.1a0e5fa80Fixes race conditions in the tx queue, where sometimes we wouldn't properly remove externalized TXs.These changes made a significant impact on top line performance. I was mostly concerned with benchmarking the phase 1 SAC goals (3k SAC TPS, 10 orgs, 2s block). We are still quite far from this goal in overlay-v2-shared, so I ran latency tests at 1k TPS and 2k TPS (3k TPS just fails at any block time). This was all with 200 ms simulated apply.
1000 TPS
2000 TPS
These changes did have an impact on median latency, but most importantly, we are much more stable under load with more bounded extremes.
Note that the diff is large because I had to self vendor a dependency. The low-level QUIC features aren't configurable based on the wrapper that libp2p exposes. It's unfortunate, QUIC supports the features, the libraries that libp2p uses under the hood support the features, but libp2p itself doesn't pass the args forward. The hack is to self-vendor a patched libp2p. Not a great prod solution, but I don't really care for this feature branch, and the features made a very meaningful impact.
Checklist
clang-formatv8.0.0 (viamake formator the Visual Studio extension)