Audit and comprehensive stability improvements in py-libp2p - #1411
Conversation
…timeout to 600s - QUICStream.read() was silently returning b"" when the stream received FIN and the receive buffer was empty. This caused read_exactly() to loop 100 times before raising IncompleteReadError, ultimately manifesting as 'fail to read from multiselect communicator' on every QUIC handshake. Fix: define _QUICStreamEOF(IOException, EOFError) locally and raise it immediately when buffer is empty and _read_closed=True. - Increase default QUIC idle_timeout from 30s to 600s (10 minutes) to match go-libp2p's default. 30s was too short: DHT queries finish quickly then both sides go idle and aioquic drops the connection before the next query arrives. - Remove debug print() calls added during investigation from QUICStream and MultiselectCommunicator.
…d fix proxy dial blocking
- The AutoConnector was previously dialing exactly `needed` peers per interval. - In a P2P network like IPFS, dial failure rates are usually > 80% due to offline nodes, NAT, etc. - We now overdial (needed * 5) and increased the capacity limiter to 50 to aggressively acquire connections and reach the low_watermark much faster.
…t ip4/ip6 The value_for_protocol method raises ProtocolLookupError when the protocol is not found in the multiaddr, instead of returning None. This caused test failures for addresses like /ip6/::1/tcp/46499 and /tcp/8080.
…et_ip_value pattern Using 'except Exception' instead of 'multiaddr.exceptions.ProtocolLookupError' since pyrefly can't resolve the 'exceptions' submodule on the multiaddr module.
The test_examples_use_address_paradigm test requires networking examples to use both get_available_interfaces and get_optimal_binding_address. The ping example was updated to use get_optimal_binding_address for displaying the optimal connection address.
51559b1 to
fa3b09a
Compare
…cks_v100 and blocks_v110 create_message does not support providing both blocks_v100 and blocks_v110 simultaneously. The test was updated to only use blocks_v100.
… type The conns attribute was changed from dict[str, float] to dict[int, float] to match the type annotation. Updated the test assertion to use id() for the key.
- peerdata: revert first_supported_protocol return to '' (matches abc.py contract and go-libp2p convention; no callers use 'None supported') - multiaddr_utils: remove DNS fallback from extract_ip_from_multiaddr (function extracts IPs, not hostnames; fixes 3 tests at once and prevents incorrect peer scoring in gossipsub) - io/utils: add early return for n=0 in read_exactly (zero-byte reads are valid per Python stream contract; fixes noise empty bytes test) - test_connection: update test_connection_start_closed for new silent shutdown-race handling in QUICConnection.start() - mplex tests: increase sleep times and add wait_all_tasks_blocked() for timing-sensitive close/reset propagation tests
Adds --durations=20 --durations-min=1.0 to pytest commands so that the 20 slowest tests (taking 1s+) are always reported in CI logs. This helps track test performance regressions.
The dedicated nursery per connection caused deadlock because trio-websocket's background reader/writer tasks keep the nursery alive indefinitely, preventing trio cleanup and causing all 8 websocket integration tests to timeout.
9f41097 to
244ef3a
Compare
… pb2 changes, stale interop files
244ef3a to
a6c5ffa
Compare
|
During the audit, I found and fixed 48 issues across 5 modules:
All fixes verified in the current codebase and tracked in the linked issues above. I’ve run the interoperability tests with the changes introduced in this PR, and all tests are now passing, as demonstrated in libp2p/unified-testing#117. |
…or audit fixes - Wrap global metrics receive loop in try/except to prevent single event failures from crashing the entire metrics collection task - Add newsfragments for issues libp2p#1390, libp2p#1413, libp2p#1414, libp2p#1415, libp2p#1416
|
This is a key milestone for py-libp2p, and I wish to thank @sumanjeet0012 for the depth and breadth of work that has gone into this PR. Also, appreciate the regular coordination with me and following up on key enhancements. The architectural audit and the resulting stability improvements touch some of the most critical parts of the stack: Bitswap, Ping, Identify/IdentifyPush, QUIC/transports, Swarm/resource management, and Kademlia DHT. The focus on resource exhaustion, memory and stream leaks, concurrency/race conditions, cancellation, protocol correctness, and closer go-libp2p interoperability makes this a particularly important step for the maturity and reliability of py-libp2p. The fact that the identified issues have been tracked through the corresponding audit issues, interoperability tests are passing, and the PR currently has all checks passing with no conflicts gives us a strong basis for moving toward final review. Given the scope and significance of this work, wish to encourage @sumanjeet0012 to open a detailed discussion page on each of the improvements made and issues resolved, particularly around the architectural changes, resource lifecycle/cancellation paths, protocol interoperability, and any potential regressions. @sumanjeet0012, excellent work on taking this audit through such a broad set of issues and bringing the branch to this stage. This PR is indeed ready for merge. We will be doing a new py-libp2p release this week and update at the blog. CCing @acul71, @pacrob, @johannamoran, @mhchia, @momack2 and @mishmosh. |
2 participants
Description
This PR is the result of a deep-dive architectural audit of the
py-libp2pstack. It contains a comprehensive series of bug fixes, architectural overhauls, and optimizations (specifically ported to thefix-1390branch).The primary goal of this audit and the resulting changes is to resolve resource exhaustion, fix memory and stream leaks, improve network stability under heavy load, and achieve closer architectural parity and interoperability with
go-libp2p.Related Issues
Below is a module-by-module breakdown of the issues identified during the audit and the improvements implemented:
1. Bitswap (Phases 1-4 Architecture Overhaul) — #1416
go-libp2p), providing isolated retrieval states and better concurrent fetching capabilities.PeerManagerandBlockPresenceManagerfor efficient tracking of peer wantlists and block availability.WANT_CANCELpropagation to peers once blocks are received, drastically saving bandwidth._pending_requests,_dont_have_responses,_expected_blocks, and_have_confirmed._pending_requestswhen fetching the same CID concurrently.trio.CancelScopeto the bitswap client loop to fix lingering task leaks on shutdown.have_blockcorrectness, added structural CID validation, and enforced block size limits.2. Ping Protocol — #1414
PingServiceto cache and reuse a single outbound stream across multipleping()calls, avoiding massive stream churn.RESP_TIMEOUTfrom 60s to 10s (matchinggo-libp2p), added write timeouts (wrappingstream.writeintrio.fail_after), and addedCancelScopesupport to abort stuck pings.AsyncIterator[int](async generator) to match the channel semantics ofgo-libp2p.read_exactly, and replaced streamclose()withreset()on payload mismatches.3. Identify & IdentifyPush — #1415
_identify_inflight._UNPARSEABLE_ADDRS_CACHE) memory leak and corrected the use ofid(conn)as a dict key to prevent stale observations on reused memory addresses.trio.Semaphoremutable default argument issue and removed a 5s synchronous poll delay that compounded stream exhaustion under load.4. QUIC & Transports — #1390
ConnGarbageCollected(0x1005 GO_AWAY) error to remote peers aggressive-trimming idle connections. Implemented application-layer keepalive streams (transport-level ping frames) for TCP and QUIC.idle_timeoutto 600 seconds to prevent premature drops.is_client=Trueconfiguration when dialing.StreamEOFcorrectly on empty buffer reads.5. Swarm & Resource Manager (rcmgr)
BasicHostto ping active connections periodically and sweep zombie connections, preventing the node from hitting its max limit with dead weight.BaseExceptionto closepre_scopeontrio.Cancelled, fixing resource leaks during outbound dialing interruptions.GRACE_PERIODto 600s, and added a floor to the graceful degradation mechanism to prevent permanent degradation loops.6. Kademlia DHT & Random Walk — #1413
min_refresh_threshold = 4) that was preventing the DHT from continuously discovering peers, causing it to endlessly redial the same peers.go-libp2p.dnsaddrresolution.7. Miscellaneous Fixes
substrinstead ofLIKEfor BLOB prefix matching, drastically improving prefix search correctness.varintusage and fixed a_cid_to_hexparsing bug.mypyandruff/pyreflyerrors, standardizing Python MRO (e.g.,_QUICStreamEOFinheritingMuxedStreamEOF).