Skip to content

S44 CI: score D-6 coverage from merged DA: lines, not the double-counted LF:/LH: summary - #254

Open
hyperxpro wants to merge 3 commits into
mainfrom
s44-coverage-metric
Open

S44 CI: score D-6 coverage from merged DA: lines, not the double-counted LF:/LH: summary#254
hyperxpro wants to merge 3 commits into
mainfrom
s44-coverage-metric

Conversation

@hyperxpro

Copy link
Copy Markdown
Member

Problem

The D-6 coverage gate is a coin flip — 3 RED / 3 green on effectively identical production source:

run h2_proxy.rs gate
28334977156 (art 7938582116) 79.60% RED
28334977156 (art 7938395373) 79.70% RED
28336426614 80.18% green
30745595161 79.65% RED
30749813681 80.23% green
30751410142 80.13% green

main is green today by luck. This also corrects two earlier records: the handoff called it "~1 run in 4" red, S43 called it "three of four samples fail" — with six samples it is ~50/50.

Cause

Re-verified independently from the RED artifact rather than inherited from the S43 write-up. lb-l7 is compiled twice and llvm-cov merges both instantiations into one SF: record whose LF:/LH: summary double-counts shared source lines:

h2_proxy.rs  declared LF/LH : 1887/1503  -> 79.65%   <- what the gate read
             distinct DA:   : 1780, hit 1441 -> 80.96%
             FN: entries    : 596  (421 CsdExPruU9iqX_5lb_l7 / 175 CsbnMibX7jh97_5lb_l7)

The second instantiation is the lib-unit-test build, which cannot reach the request hot path at allhyper::body::Incoming has no public constructor — so its unhit lines were scored as genuine misses. This is systemic: 29 hot-path modules are dual-instantiated; h2_proxy was just the only one sitting on the boundary.

Fix

Score merged DA: records — each source line counted once, hit if any instantiation ran it. Threshold stays 80.0. Patterns unchanged. The single named loader.rs carve-out unchanged. Nothing waived or exempted.

This is a correction, not a relaxation — and the re-baseline proves it

Of 31 hot-path modules: 9 up, 12 DOWN, 10 unchanged, 0 below 80%.

A relaxation is a one-way ratchet — it cannot make random.rs (86.21→85.71), conn_gate.rs (91.14→90.91) or conn_actor.rs (85.01→84.79) stricter, which this does, because double-counted hit lines leave the numerator too. Full 31-row table in audit/ci/s44-coverage-metric-rebaseline.md.

Two latent defects fixed in passing

  • the old parser did files[cur] = … per record, so multiple SF: records for one file were last-record-wins;
  • writing the merge as if cnt > d.get(ln, 0) is a trap — 0 > 0 is False, so a line hit by no instantiation never enters the dict, silently leaves the denominator, and every file scores 100%. Caught only because the expected value (80.96%) was known in advance; a pass/fail-only check would have shipped it green.

Verification

Every value cross-checked against an independently written scorer, not the gate's own output.

sample old → new gate
red 30745595161 79.65% → 80.96% exit 0
green 30749813681 80.23% → 81.57% exit 0
green 30751410142 80.13% → 81.52% exit 0

Load-bearing negative controls — the gate must still fail when coverage genuinely drops:

control expect result
zero 576 of 1441 hit lines RED 48.54%, exit 1 ✅
1424/1780 (exactly 80.00%) pass exit 0 ✅
1423/1780 (79.94%) RED exit 1 ✅
all paths renamed fail closed exit 1 ✅
empty LCOV fail closed exit 1 ✅
carve-outs exactly 1

Single-line discrimination at the floor — the gate is non-vacuous.

What this does NOT fix

The ~0.6 pp run-to-run jitter is untouched; only the level moves. h2_proxy now sits ~1.0–1.6 pp above the floor instead of straddling it — a real margin, not a comfortable one. If it drifts down again the honest fix is real coverage (S43 option (c)), not another metric change.

Follow-on

With this landed, Coverage (per-module hot-path >= 80%) becomes safe to add to required status checks. Before it, requiring that check would have blocked ~half of all PRs on a measurement artifact. Owner copy-paste pack for branch protection + the SOAK_* values: audit/release/owner-actions.md.

…ted LF:/LH: summary

The D-6 gate was a coin flip: 3 RED / 3 green on effectively identical
production source (79.60 / 79.70 / 80.18 / 79.65 / 80.23 / 80.13 on
h2_proxy.rs). main was green at handoff by luck. This corrects two earlier
records -- the handoff called it "~1 run in 4" red, S43 called it "three of
four samples fail"; with six samples it is ~50/50.

Cause, re-verified independently from the RED artifact rather than inherited
from the S43 write-up: lb-l7 is compiled twice and llvm-cov merges both
instantiations into ONE SF: record. Its LF:/LH: summary double-counts shared
source lines (h2_proxy declared LF:1887 while emitting 1780 distinct DA:
lines -- 107 phantom) and carries 596 FN: entries split across two crate
disambiguators. The second instantiation is the lib-unit-test build, which
cannot reach the request hot path at all because hyper::body::Incoming has no
public constructor, so its unhit lines were scored as genuine misses. This is
systemic -- 29 hot-path modules are dual-instantiated; h2_proxy was just the
only one on the boundary.

Score merged DA: records instead: each source line counted once, hit if ANY
instantiation ran it. Threshold stays 80.0. Patterns unchanged. The single
named loader.rs carve-out is unchanged. Nothing waived or exempted.

This is a CORRECTION, not a relaxation, and the re-baseline proves it: of 31
hot-path modules, 9 go up, 12 go DOWN, 10 are unchanged, 0 fall below 80%. A
relaxation is a one-way ratchet -- it cannot make random.rs, conn_gate.rs or
conn_actor.rs stricter, which this does, because double-counted HIT lines
leave the numerator too.

Two latent defects fixed in passing:
  * the old parser did files[cur] = ... per record, so multiple SF: records
    for one file were last-record-wins;
  * writing the merge as `if cnt > d.get(ln, 0)` is a trap -- 0 > 0 is False,
    so a line hit by no instantiation never enters the dict, silently leaves
    the DENOMINATOR, and every file scores 100%. Caught only because the
    expected value (80.96%) was known in advance; a pass/fail-only check
    would have shipped it green. Uses max() now, with a comment saying why.

Verified against all three real CI artifacts, every value cross-checked
against an independently written scorer. Load-bearing negative controls:
zeroing 576 of 1441 hit lines -> 48.54% RED; 1424/1780 -> 80.00% pass while
1423/1780 -> 79.94% RED (single-line discrimination at the floor); renamed
paths and empty LCOV both fail closed; exactly one carve-out remains.

Does NOT fix the ~0.6pp run-to-run jitter -- only the level. h2_proxy now
sits ~1.0-1.6pp above the floor instead of straddling it. If it drifts down
again the honest fix is real coverage, not another metric change.

Full re-baseline table: audit/ci/s44-coverage-metric-rebaseline.md
Owner copy-paste pack for branch protection + SOAK_*: audit/release/owner-actions.md
proto added 2 commits August 2, 2026 16:03
… cause)

The rebaseline doc said the ~0.6pp run-to-run jitter was "not investigated".
It has now been tested against the obvious candidate and refuted, so the next
session does not re-chase it.

Hypothesis: the coverage job runs `cargo llvm-cov nextest --ignore-run-fail`,
so a flaking test still counts as executed and its lines could drop out of the
profile -- and there is a known flake named for this exact job and module
(CF-S37-D6-H2PROXY-FLAKY).

Refuted by the nextest summaries inside the three coverage runs:

  30745595161 RED    0 failed (1565/1565)  -> 79.65%  <- LOWEST
  30749813681 green  2 failed              -> 80.23%
  30751410142 green  0 failed (1565/1565)  -> 80.13%

The run with zero failures produced the lowest coverage and the run with two
failures the highest. The correlation runs opposite to the hypothesis.

Mechanism remains unknown. Recorded the untested candidates and the suggested
next step (diff the per-line DA: sets between two runs to see WHICH lines
move, rather than reasoning from totals).
…pc_h3_e2e

Found while verifying the coverage-metric change. The Coverage job ran 5+
hours with no progress and had to be cancelled; it would otherwise have sat
until GitHub's 6-hour job timeout.

nextest's escalating SLOW markers and the orphan-process sweep name it:

  SLOW [>17940.000s] lb-quic::grpc_h3_e2e grpc_h3_without_te_header_still_delivers_trailer
  Terminate orphan process: pid (12606) (grpc_h3_e2e-ee1b7dc6f6c6ee7e)

1564 of 1565 tests completed; that one never returned -- ~17,940s against the
test's own OVERALL = 20s budget, roughly 900x over. It is a hang, not slowness.

NOT caused by the S44 coverage change: coverage-check.sh runs in the NEXT
workflow step, which was still pending and never executed, and the change is a
pure LCOV text parser that cannot affect a QUIC test binary.

Worse than a failing test, because the job runs
`cargo llvm-cov nextest --ignore-run-fail`: a hang is not a failure, so the
flag cannot help and the job neither fails nor completes -- it burns a runner
slot silently. There is no timeout-minutes: on the job, so the ceiling is the
6-hour default. The uninstrumented Test job passed in the same run, so the
Test lane is not a reliable detector.

Two candidate mechanisms are recorded and DELIBERATELY not chosen between,
because the log cannot separate them: an untimed SUITE_SERIAL.lock().await in
the harness (one stuck test then wedges the whole binary), or the deadline not
bounding every await path (setup runs outside the budget entirely). The
structural point stands either way.

Re-run on the identical SHA went 16/16 green in 986s with the hanging test
passing in 0.100s, so the hang is intermittent -- but that same re-run failed a
DIFFERENT trailer test in the SAME binary:

  grpc_h3_trailer_survives_all_response_sizes
  assertion left == right failed: sz=262144  left: Some(502)  right: Some(200)

A 502 instead of 200 at a 256 KiB response, not on the known-flake list. Both
occurrences are instrumentation-only. Whether that 502 is a tight test-backend
timeout or a genuine gateway failure on large H3 responses is OPEN, and the
second possibility is a product bug, so it is not assumed away.

Recommends containment (timeout-minutes:, bounded serial guard, bounded test)
while being explicit that containment is not a fix, and that the test must not
be deleted or #[ignore]d -- it covers real protocol behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant