Sanitizer (ASan/TSan) coverage gaps in //flare/fiber/... beyond the switch fixes

[chen3feng]

Surfaced while adding the ASan/TSan fiber CI lane (#198). The fiber-switch SEGVs are fixed (#196 ASan, #197 TSan) and a curated set of fiber tests now runs green under both sanitizers. Running the full //flare/fiber/... under sanitizers, however, reveals separate pre-existing issues that #198 intentionally left out of scope (so the lane stays green). Tracking them here.

Environment: Bazel + clang, --config=asan / --config=tsan (see .bazelrc). Observed on clang-17; the annotation.h TSan-header build break is already fixed in #198.

1. curl fails to build under -fsanitize

curl (foreign_cc / autotools subbuild) fails when the sanitizer flags are propagated into its build:

ERROR: .../external/com_github_curl_curl/BUILD.bazel: Foreign Cc - Configure: Building curl failed
==NNN==ERROR: AddressSanitizer: use-after-poison ...

Consequence: every fiber target that transitively depends on curl fails to build under ASan/TSan, e.g. execution_context_test, execution_context_benchmark, fiber_local_benchmark, logging_test. Likely needs the sanitizer copts excluded from foreign_cc external builds (curl isn't what we're testing).

2. Test failures under sanitizers

These build fine but fail when run under sanitizers (need investigation — could be real data races / UB, or test-harness issues):

test	ASan	TSan
//flare/fiber:latch_test	FAILED	(pass)
//flare/fiber/detail:scheduling_group_test	FAILED	FAILED
//flare/fiber/detail:waitable_test	FAILED	TIMEOUT

3. Tests too slow / timing out under TSan

Under TSan's slowdown on shared runners, some tests exceed even a raised timeout: //flare/fiber:fiber_test, //flare/fiber/detail:run_queue_test (and fiber_worker_test is ~5 min). These are excluded from the CI lane for now; they'd need a larger timeout and/or to be made lighter.

Why not in the CI lane

The #198 sanitizer_fiber job runs a curated subset that passes cleanly under both sanitizers — enough to guard the switch fixes (a regressed switch crashes every fiber test on first resume). Expanding it to the full suite requires resolving the above first.

🤖 Generated with Claude Code

[chen3feng]

Triaged the "test failures" by running each in isolation (ASan, clang-17). The classification in the issue body was too coarse — refined:

• scheduling_group_test — not a bug, a timeout. It's a stress test (Starting 131072 fibers, ~25s per parametrization × 10) that exceeds the test timeout under sanitizer slowdown (and trips object-pool high-water-mark churn). Same bucket as the "too slow under TSan" tests — needs a larger timeout and/or a smaller fiber count under -fsanitize.

• latch_test — flake, not a real failure. Passes cleanly in isolation (4/4). Its full-suite "FAILED in 0.6s" was a startup failure (too fast to be a test assertion) — resource contention when ~29 sanitizer test processes ran concurrently.

• waitable_test — REAL ASan finding. Reproducible in isolation:

  ERROR: AddressSanitizer: use-after-poison ... WRITE of size 16 ... thread T88
  in Waitable/SystemFiberOrNot.Mutex/0
  

  Likely flare's manual ASan poisoning (fiber stack / object pool via __asan_(un)poison_memory_region) not being reconciled when the fiber mutex reuses that memory — or a genuine use-after-poison. The full stack trace is hidden by ASan's "nested bug in the same thread, aborting" (the report handler re-faults), so it needs a gdb dig like clang-17 asan/tsan fiber coredump #134's TSan trace.

So the actionable items here are really: (1) exclude third-party foreign_cc libs (curl, …) from -fsanitize so curl-dependent targets build; (2) investigate the waitable_test use-after-poison; (3) give the heavy stress tests bigger timeouts / smaller counts under sanitizer.

[chen3feng]

Deep-dive: the waitable_test / scheduling_group_test ASan failures are a fiber-stack poison race

Root-caused the use-after-poison (it's the same class in both tests). Reproduces reliably (~1st attempt) with ASAN_OPTIONS=detect_stack_use_after_return=0:fast_unwind_on_fatal=1 (the default slow unwind re-faults on the poisoned stack → "nested bug in the same thread, aborting", which is why the trace was hidden).

Stack trace (waitable Mutex):

WRITE of size 16 ... thread T1038
  #4  tls::detail::LazyInitObjectArray::UninitializedExpand()   object_array.h:100  (placement-new / vector<bool> resize)
  ...
  #13 WriteMostly::Update()                                     write_mostly.h:44
  #15 SchedulingGroup::WakeUpOneDeepSleepingWorker()            scheduling_group.cc:599
  #18 SchedulingGroup::ReadyFiber()                             scheduling_group.cc:370
  #19 fiber::detail::Mutex::unlock()                            waitable.cc:208

ASan address description:

Address ... is located in stack of thread T1038 at offset 64 in frame
Shadow byte legend ... Poisoned by user: f7

So the written-to memory is a fiber stack that flare poisoned (f7 = __asan_poison_memory_region, done by stack_allocator.h's OnPut when a stack returns to the object pool), being written while the fiber is still running. scheduling_group_test's SetTimerPeriodic abort is the same mechanism.

scheduling_group_test isn't a fiber-count problem (capping 131072→4096 just makes it reach the abort faster) and latch_test is unrelated (a startup flake under concurrent load).

Why: flare keeps pooled fiber stacks poisoned to catch use-after-free of a stack (doc/sanitizers.md). The flip side is exactly this: any path that touches a stack around the pool put/get boundary trips use-after-poison. The doc itself notes the cleaner approach: "理论上来说，更好的做法是在 ASan 环境下从不复用栈".

Proposed fix direction: under FLARE_INTERNAL_USE_ASAN, don't pool/reuse fiber stacks (allocate fresh + unpoison on free, no OnPut re-poison) — eliminating the poisoned-while-reachable window. Costs stack-alloc churn under ASan only; that's acceptable for a sanitizer build, and it's what the doc already recommends. Needs verifying it clears both waitable_test and scheduling_group_test under ASan.

[chen3feng]

Fix attempt — two hypotheses experimentally disproven; root cause is deeper

Tried to fix the use-after-poison and verified each under ASan (clang-17):

1. Removed flare's manual fiber-stack poisoning (stack_allocator.h Create/OnGet/OnPut/Destroy) → waitable_test still use-after-poison, identical spot (the std::vector<bool> op in the TLS ObjectArray). So the poison is not coming from flare's __asan_poison_memory_region on pooled stacks. (Reverted this change — it removed real detection for no gain.)
2. -fno-sanitize-address-use-after-scope → still aborts. So it isn't stack-use-after-scope instrumentation either.

Refined hypothesis: the shadow is f7 ("poisoned by user") and flare's only manual-poison site is the stack allocator (ruled out in #1). The remaining f7 source is ASan's own fiber-switch machinery — __sanitizer_start_switch_fiber / __sanitizer_finish_switch_fiber poison a fiber's stack while it's considered "switched away". The crash path (Mutex::unlock → WakeUpOneDeepSleepingWorker → a WriteMostly metric writing to a fiber-stack frame) suggests a window where a stack ASan considers suspended is written to. That puts this in the same family as #134 (the StartSwitchFiber/CompleteSwitchFiber annotations in flare/base/internal/annotation.h + fiber_entity.cc), i.e. the core ASan fiber-switch support — not the object pool.

This is a genuine, deep, non-deterministic bug in the ASan fiber-switch annotations under concurrent scheduling. It resists config/flag-level fixes (two ruled out) and needs dedicated work on the switch-fiber stack-bounds/ordering. Documenting here for whoever picks it up; the reliable repro is ASAN_OPTIONS=detect_stack_use_after_return=0:fast_unwind_on_fatal=1 bazel-bin/flare/fiber/detail/waitable_test (~1st run).