Android uploads can stall after UDP segmentation-offload send failures

[momics]

This was generated by AI during triage.

What happened

An Android-to-desktop interop run did not complete and appeared to fail around the 4 KiB and 64 KiB request-body cases. The serving desktop recorded multiple 30-second body-frame read timeouts and one incomplete request body.

A subsequent run between the same peers completed all 107 cases with 106 passes, no failures, and one expected skip. Every request-body case from 1 byte through 5 MiB passed on the rerun.

What I expected

The Android-to-desktop request-body checks should complete deterministically on the first run. A transient incomplete body or stalled body frame should either recover cleanly or produce a bounded, actionable failure without preventing the suite from completing.

Steps to reproduce

1. Start fresh Android and desktop test peers on the same network.
2. Enable testing mode and select the desktop peer on Android.
3. Run the complete Android-to-desktop interop suite.
4. Observe whether the suite stalls or fails while entering the request-body size cases.
5. Rerun the same suite without changing peers and compare the result.

Additional context

Observed with Android 16 / API 36 against desktop using an exact-main development build. Generic DNS-SD browse, advertise, removal, and restart checks all passed during the same session. The clean rerun used a measured direct transport path.

[momics]

This was generated by AI during triage.

Reproduced again on exact main (1f85f4ad5efcd141cbb0bc6e0fb15c18d2742c0b) with a freshly started desktop peer and a cold-relaunched Android 16 / API 36 app.

The submitted Android-to-desktop report completed all 107 cases with 105 passes and 2 failures:

• http-compliance-body-size-4kb: hit the 20-second case deadline (20,003 ms)
• http-compliance-body-size-64kb: hit the 20-second case deadline (20,010 ms)

The immediately following body cases recovered without restarting either peer:

• 1 MiB passed, but took 15,697 ms
• 5 MiB passed in 723 ms

The serving desktop logged body frame read timed out after 30s during the same interval. For comparison, an earlier warm rerun completed the same body cases in approximately 27 ms, 64 ms, 241 ms, and 840 ms respectively.

This confirms a real intermittent body-stream/pump stall. It is not the already-fixed suite-progress presentation issue, and the passing 5 MiB case rules out a simple request-size ceiling.

[momics]

This was generated by AI during triage.

Two immediate warm reruns provide a tighter diagnosis:

Run	Failed body case	Following recovery
Cold	4 KiB and 64 KiB	1 MiB took 15.7 s; 5 MiB took 0.7 s
Warm 1	64 KiB	1 MiB took 3.5 s; 5 MiB took 0.7 s
Warm 2	1 MiB	5 MiB took 7.3 s

Each warm-run failure was immediately preceded on Android by a UDP sendmsg failure with OS error 5 (EIO) for a multi-segment transmit. The selected direct path was subsequently abandoned and the connection moved to relay, after which the next large-body case recovered. The failed size moving from 64 KiB to 1 MiB also confirms that the payload size itself is not the trigger.

This matches the failure mode addressed by upstream noq PR n0-computer/noq#746, “Retransmit failing GSO packets and split batches.” Its description specifically calls out Android, and its patch retries a failed multi-segment GSO send as individual datagrams. The released dependency currently used here (iroh 1.0.2 / noq 1.0.1) does not contain that open fix.

Current assessment: this is an Android UDP/GSO fallback defect below the HTTP body bridge. The HTTP suite exposes it reliably because the larger uploads exercise multi-segment sends. A focused confirmation would be to disable QUIC segmentation offload on Android through QuicTransportConfigBuilder::enable_segmentation_offload(false), rebuild, and rerun the same matrix. That should remain a bounded Android workaround and be removable once the upstream fix ships.

[momics]

This was generated by AI during triage.

After leaving the same Android and desktop processes running, two subsequent foreground runs completed cleanly with 107/107 passes.

Body timings were stable on both runs:

Case	Pass 1	Pass 2
4 KiB	27 ms	20 ms
64 KiB	57 ms	43 ms
1 MiB	220 ms	205 ms
5 MiB	802 ms	802 ms

This does not invalidate the earlier failures from the unchanged build. It strengthens the stateful transport diagnosis: after the Android connection/path has recovered and stabilized, the HTTP and FFI body path performs normally. The release question remains whether disabling segmentation offload on Android prevents the initial/intermittent EIO path degradation.

[momics]

This was generated by AI during triage.

The Android-only transport diagnostic has now completed successfully against exact main.

Change under test:

• disable QUIC segmentation offload in the iroh transport configuration on Android only
• retain the existing upstream default on iOS, macOS, Linux, and Windows
• keep the workaround explicitly linked to [noq-udp] Retransmit failing GSO packets and split batches n0-computer/noq#746 for removal once that fix is released

Device evidence:

• 3 consecutive Android -> desktop full suites
• every run: 107 pass / 0 fail / 0 skip
• 4 KiB: 29.9-32.2 ms
• 64 KiB: 39.8-82.5 ms
• 1 MiB: 227.3-251.1 ms
• 5 MiB: 796-919.3 ms

Transport-log interpretation:

• no QUIC application-data send errors occurred during the three runs
• one 60-byte QADv4 connectivity probe still encountered Android's kernel GSO rejection; that probe uses iroh's separate network-report path, immediately halted its own segmentation offload, and did not affect any suite case
• this is consistent with the workaround's intended scope: prevent batched QUIC data packets from being lost while leaving the independent connectivity probe under upstream ownership

Local verification:

• all 97 iroh-http-core library tests pass
• focused Android/non-Android policy tests pass
• normal strict library clippy passes
• a fresh embedded Android APK built and installed successfully

Upstream noq#746 remains open, so the local workaround should remain temporary and be removed after the fixed noq release is adopted.