Parallel-execution flakes in the codewhale-tui lib suite (tracking): six tests that fail under load and pass in isolation

[Hmbown]

Every one of these failed exactly once tonight (2026-09-06) in a full RUST_MIN_STACK=16777216 cargo test -p codewhale-tui --lib run or a CI leg, passed in isolation, and passed on the next full run. None was touched by the PR that hit it. Each costs a rerun or a 25-minute matrix.

Test	Where seen	Symptom
hooks::executor::tests::background_hook_receives_the_same_stdin_payload_as_foreground	#5921 ubuntu-latest	valid JSON: EOF while parsing a value at executor.rs:4649 — the background hook read an empty stdin
session_control_acceptance::tests::tui_and_api_listings_agree	#5920 local full run	touches shared-home session files
remote_control::tests::separate_predispatch_crashes_on_one_run_get_distinct_recovery_turn_ids	#5921 and #5924 local full runs (also fails on unmodified main under load)
model_inventory::…ollama_default_prefers_live_local_tags…	#5924 local full run	live local tags probe under load
client::tests::deepseek_anthropic_translate_uses_messages_endpoint	#5920 local full run
tui::clipboard::tests::tmux_load_buffer_w_reaches_attached_client_with_default_passthrough_disabled	#5920 local full run

Related, already fixed or filed: #5898 (idle-timeout race, fixed by #5913), #5919 (/pin blocks on the Windows message pump, #5922).

Acceptance: each test either isolates its shared state (temp home, no wall-clock races, no live probes) or is marked with the reason it cannot run in parallel; three consecutive green full-suite runs under cargo nextest default parallelism on macOS and ubuntu.

[Hmbown]

Two more from tonight's full-suite runs on the #5935 branch, both passing in isolation and not touched by that PR:

• runtime_api::tests::steer_and_interrupt_endpoints_work_on_active_turn — this one was real once: the first cut of fix(tui): show the snapshots-off notice in the TUI and name the approval log on receipt failures #5935 drained a process-wide notice queue, so another test's parked notice surfaced as an unexpected event. Fixed by keying the drain per workspace; listed here because the same shape (global static shared across tests) is worth grepping for.
• runtime_threads::tests::engine_error_remains_failed_after_nominal_turn_complete — failed once in a full run, 1/1 in isolation.

[Hmbown]

One more from a full-suite run on the #5961 branch, unrelated to its files and passing alone: runtime_chat_relay::tests::failed_state_writes_never_become_in_memory_authority_and_exact_retry_reopens.

[Hmbown]

From full-suite runs on the #5964 and #5963 branches (unrelated files, pass in isolation): runtime_api::tests::compatibility_stream_* (two tests, "deadline has elapsed" under parallel load) and, again, remote_control::tests::separate_predispatch_crashes_on_one_run_get_distinct_recovery_turn_ids and model_inventory::tests::ollama_default_prefers_live_local_tags_over_the_unresolved_marker. The tracker is now at eleven distinct tests; the runtime_api deadline pair looks like a fixed timeout that starves under 14k parallel tests.

[Hmbown]

Root-cause analysis for the six tracked tests, from #5970. Five fixed there; one not reproduced, with the evidence below.

The five fixed

1. hooks::executor::tests::background_hook_receives_the_same_stdin_payload_as_foreground — TOCTOU on the capture file, not an executor defect. The hook script redirects with > out, so the shell creates out empty at exec and cat writes the payload afterwards; the test only polled for out.exists(). Under load the read won that race and serde_json::from_str("") produced exactly the recorded valid JSON: EOF while parsing a value. Fix: wait for non-empty bytes (the payload is a single small write). The identical sibling background_hook_receives_the_documented_environment had the same race and got the same fix.

2. session_control_acceptance::tests::tui_and_api_listings_agree — the test redirected process-global CODEWHALE_HOME to a temp home and then read the store twice: its own list_sessions() snapshot plus the picker's construction-time default_location() listing. Any concurrent test that resolves default_location() without the env barrier (several call paths do) could observe the redirect mid-test and write a session into the fixture between the two reads, breaking picker==API equality (and polluting that test's own store in the opposite direction). Fix: private SessionManager on a temp dir plus a #[cfg(test)] picker seam (new_with_sessions) that hands the picker the same list — no env mutation at all, the picker's filter/sort/scope pipeline still fully exercised.

3. model_inventory::tests::ollama_default_prefers_live_local_tags_over_the_unresolved_marker — the "live local tags" were real. A concurrent switch_provider(..., ApiProvider::Ollama, ...) (e.g. tui::ui::tests::provider_switch_clears_turn_cache_history) fires spawn_active_provider_catalog_refresh, whose detached tokio task probes the real local daemon at 127.0.0.1:11434 and merges whatever it answers into the process-wide provider lake — with no synchronization against provider_lake::lock_live_snapshot (which only serializes tests that take it). An Ollama daemon was confirmed listening on this machine while debugging, answering real tags, which then flip the test's empty-snapshot "unknown" assertion. Fix: spawn_active_provider_catalog_refresh is a no-op under cfg(test); catalog refresh remains covered by the stubbed-endpoint fetch_catalog_delta_* / refresh_catalog_cache_* tests. This also stops the unit-test binary from ever hitting a developer's local daemon.

4. client::tests::deepseek_anthropic_translate_uses_messages_endpoint — an env race, not a wire problem. translate resolves max_tokens (reading CODEWHALE_MAX_OUTPUT_TOKENS/DEEPSEEK_MAX_OUTPUT_TOKENS) when building the request, and the test recomputes the same route allowance when asserting. Several concurrent tests legitimately mutate those vars under lock_test_env (outbound_seam_*, matched_vision_route_*, core::engine tests, effective_max_output_tokens_env_override_*); landing between the two reads makes request value and assertion value disagree. Fix: the translate test holds lock_test_env() for its whole body so both reads observe one stable environment.

5. tui::clipboard::tests::tmux_load_buffer_w_reaches_attached_client_with_default_passthrough_disabled — the two fixed 3s wall-clock bounds (client attach, OSC 52 receipt) starve on a machine running a full parallel suite; same family as the runtime_api::compatibility_stream_* "deadline has elapsed" pair already noted in this tracker. Fix: bounds raised to 30s; what the test verifies is unchanged.

The sixth — not root-caused

remote_control::tests::separate_predispatch_crashes_on_one_run_get_distinct_recovery_turn_ids — I could not reproduce it and could not find a defect by reading. By construction the test is deterministic and self-contained: the journal lives in a private tempdir; the three journal opens are strictly sequential with the flock released by Drop before each reopen; the two recovery turn ids are SHA-256 hashes over fresh UUIDv4 lease ids (assert_ne cannot fail without a UUID collision); no wall-clock input is asserted; the only process-wide test static (TEST_JOURNAL_PERSIST_FAILURES) is keyed by exact journal path; no network. Evidence: 69 isolated repetitions, 15 whole-remote_control::tests module runs at --test-threads 16, and 32 concurrent-self invocations — all green, on a machine under load. No code change was made; inventing one without a mechanism would be cargo-cult. It stays on this tracker. My best remaining suspects are environmental (transient FS errors surfacing through one of the unwrap()s on disk-pressure-constrained runners, or abort misattribution — see the next section, which produced exactly that shape locally).

Operational finding: the shared CARGO_TARGET_DIR is producing fake full-run failures

While gating #5970, two full-suite attempts failed ~2,100 tests each with panicked at crates/tui/src/config.rs: index out of bounds: the len is 48 but the index is 48. That panic is not reachable from current main's source (its lookup tables are len 49/50). Cause: multiple worktrees share CARGO_TARGET_DIR=/Volumes/VIXinSSD/CW/codewhale/target, and worktrees on older bases still have a 48- (or 43-) variant ProviderKind; because the metadata hash does not include source content, their builds clobber the same libcodewhale_config-*.rlib and even the same codewhale_tui-* test-binary path that a current-main build just wrote. A full run started in that window executes another branch's binary (the poisoned runs listed 11,872 tests instead of 11,881 — a reliable tell) and mass-fail provider-touching tests. Any "local full run" flake report from this machine is suspect until this is addressed (e.g. per-worktree CARGO_TARGET_DIR, or a workspace-status check that refuses to run suites while diverged worktrees are building). Related: running the suite without RUST_MIN_STACK (plain cargo test, outside scripts/dev-cargo.sh / CI) stack-overflows test threads and aborts the binary mid-run.

Evidence per test (on a sentinel-verified private copy of the binary built from #5970): each fixed test 12/12 consecutive green singles at --test-threads 16, its whole module 2–5× green, direct interference pairs (ollama-switch vs inventory; env-mutators vs translate) 12/12 each, and one full suite run green: 11869 passed / 0 failed / 13 ignored. The three-consecutive-full-runs acceptance bar was not met on this shared machine today; the two invalid attempts are the clobbering cases above, not test failures.