S44 CI: score D-6 coverage from merged DA: lines, not the double-counted LF:/LH: summary - #254
Open
hyperxpro wants to merge 3 commits into
Open
S44 CI: score D-6 coverage from merged DA: lines, not the double-counted LF:/LH: summary#254hyperxpro wants to merge 3 commits into
hyperxpro wants to merge 3 commits into
Conversation
…ted LF:/LH: summary
The D-6 gate was a coin flip: 3 RED / 3 green on effectively identical
production source (79.60 / 79.70 / 80.18 / 79.65 / 80.23 / 80.13 on
h2_proxy.rs). main was green at handoff by luck. This corrects two earlier
records -- the handoff called it "~1 run in 4" red, S43 called it "three of
four samples fail"; with six samples it is ~50/50.
Cause, re-verified independently from the RED artifact rather than inherited
from the S43 write-up: lb-l7 is compiled twice and llvm-cov merges both
instantiations into ONE SF: record. Its LF:/LH: summary double-counts shared
source lines (h2_proxy declared LF:1887 while emitting 1780 distinct DA:
lines -- 107 phantom) and carries 596 FN: entries split across two crate
disambiguators. The second instantiation is the lib-unit-test build, which
cannot reach the request hot path at all because hyper::body::Incoming has no
public constructor, so its unhit lines were scored as genuine misses. This is
systemic -- 29 hot-path modules are dual-instantiated; h2_proxy was just the
only one on the boundary.
Score merged DA: records instead: each source line counted once, hit if ANY
instantiation ran it. Threshold stays 80.0. Patterns unchanged. The single
named loader.rs carve-out is unchanged. Nothing waived or exempted.
This is a CORRECTION, not a relaxation, and the re-baseline proves it: of 31
hot-path modules, 9 go up, 12 go DOWN, 10 are unchanged, 0 fall below 80%. A
relaxation is a one-way ratchet -- it cannot make random.rs, conn_gate.rs or
conn_actor.rs stricter, which this does, because double-counted HIT lines
leave the numerator too.
Two latent defects fixed in passing:
* the old parser did files[cur] = ... per record, so multiple SF: records
for one file were last-record-wins;
* writing the merge as `if cnt > d.get(ln, 0)` is a trap -- 0 > 0 is False,
so a line hit by no instantiation never enters the dict, silently leaves
the DENOMINATOR, and every file scores 100%. Caught only because the
expected value (80.96%) was known in advance; a pass/fail-only check
would have shipped it green. Uses max() now, with a comment saying why.
Verified against all three real CI artifacts, every value cross-checked
against an independently written scorer. Load-bearing negative controls:
zeroing 576 of 1441 hit lines -> 48.54% RED; 1424/1780 -> 80.00% pass while
1423/1780 -> 79.94% RED (single-line discrimination at the floor); renamed
paths and empty LCOV both fail closed; exactly one carve-out remains.
Does NOT fix the ~0.6pp run-to-run jitter -- only the level. h2_proxy now
sits ~1.0-1.6pp above the floor instead of straddling it. If it drifts down
again the honest fix is real coverage, not another metric change.
Full re-baseline table: audit/ci/s44-coverage-metric-rebaseline.md
Owner copy-paste pack for branch protection + SOAK_*: audit/release/owner-actions.md
added 2 commits
August 2, 2026 16:03
… cause) The rebaseline doc said the ~0.6pp run-to-run jitter was "not investigated". It has now been tested against the obvious candidate and refuted, so the next session does not re-chase it. Hypothesis: the coverage job runs `cargo llvm-cov nextest --ignore-run-fail`, so a flaking test still counts as executed and its lines could drop out of the profile -- and there is a known flake named for this exact job and module (CF-S37-D6-H2PROXY-FLAKY). Refuted by the nextest summaries inside the three coverage runs: 30745595161 RED 0 failed (1565/1565) -> 79.65% <- LOWEST 30749813681 green 2 failed -> 80.23% 30751410142 green 0 failed (1565/1565) -> 80.13% The run with zero failures produced the lowest coverage and the run with two failures the highest. The correlation runs opposite to the hypothesis. Mechanism remains unknown. Recorded the untested candidates and the suggested next step (diff the per-line DA: sets between two runs to see WHICH lines move, rather than reasoning from totals).
…pc_h3_e2e Found while verifying the coverage-metric change. The Coverage job ran 5+ hours with no progress and had to be cancelled; it would otherwise have sat until GitHub's 6-hour job timeout. nextest's escalating SLOW markers and the orphan-process sweep name it: SLOW [>17940.000s] lb-quic::grpc_h3_e2e grpc_h3_without_te_header_still_delivers_trailer Terminate orphan process: pid (12606) (grpc_h3_e2e-ee1b7dc6f6c6ee7e) 1564 of 1565 tests completed; that one never returned -- ~17,940s against the test's own OVERALL = 20s budget, roughly 900x over. It is a hang, not slowness. NOT caused by the S44 coverage change: coverage-check.sh runs in the NEXT workflow step, which was still pending and never executed, and the change is a pure LCOV text parser that cannot affect a QUIC test binary. Worse than a failing test, because the job runs `cargo llvm-cov nextest --ignore-run-fail`: a hang is not a failure, so the flag cannot help and the job neither fails nor completes -- it burns a runner slot silently. There is no timeout-minutes: on the job, so the ceiling is the 6-hour default. The uninstrumented Test job passed in the same run, so the Test lane is not a reliable detector. Two candidate mechanisms are recorded and DELIBERATELY not chosen between, because the log cannot separate them: an untimed SUITE_SERIAL.lock().await in the harness (one stuck test then wedges the whole binary), or the deadline not bounding every await path (setup runs outside the budget entirely). The structural point stands either way. Re-run on the identical SHA went 16/16 green in 986s with the hanging test passing in 0.100s, so the hang is intermittent -- but that same re-run failed a DIFFERENT trailer test in the SAME binary: grpc_h3_trailer_survives_all_response_sizes assertion left == right failed: sz=262144 left: Some(502) right: Some(200) A 502 instead of 200 at a 256 KiB response, not on the known-flake list. Both occurrences are instrumentation-only. Whether that 502 is a tight test-backend timeout or a genuine gateway failure on large H3 responses is OPEN, and the second possibility is a product bug, so it is not assumed away. Recommends containment (timeout-minutes:, bounded serial guard, bounded test) while being explicit that containment is not a fix, and that the test must not be deleted or #[ignore]d -- it covers real protocol behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The D-6 coverage gate is a coin flip — 3 RED / 3 green on effectively identical production source:
mainis green today by luck. This also corrects two earlier records: the handoff called it "~1 run in 4" red, S43 called it "three of four samples fail" — with six samples it is ~50/50.Cause
Re-verified independently from the RED artifact rather than inherited from the S43 write-up.
lb-l7is compiled twice and llvm-cov merges both instantiations into oneSF:record whoseLF:/LH:summary double-counts shared source lines:The second instantiation is the lib-unit-test build, which cannot reach the request hot path at all —
hyper::body::Incominghas no public constructor — so its unhit lines were scored as genuine misses. This is systemic: 29 hot-path modules are dual-instantiated; h2_proxy was just the only one sitting on the boundary.Fix
Score merged
DA:records — each source line counted once, hit if any instantiation ran it. Threshold stays 80.0. Patterns unchanged. The single namedloader.rscarve-out unchanged. Nothing waived or exempted.This is a correction, not a relaxation — and the re-baseline proves it
Of 31 hot-path modules: 9 up, 12 DOWN, 10 unchanged, 0 below 80%.
A relaxation is a one-way ratchet — it cannot make
random.rs(86.21→85.71),conn_gate.rs(91.14→90.91) orconn_actor.rs(85.01→84.79) stricter, which this does, because double-counted hit lines leave the numerator too. Full 31-row table inaudit/ci/s44-coverage-metric-rebaseline.md.Two latent defects fixed in passing
files[cur] = …per record, so multipleSF:records for one file were last-record-wins;if cnt > d.get(ln, 0)is a trap —0 > 0is False, so a line hit by no instantiation never enters the dict, silently leaves the denominator, and every file scores 100%. Caught only because the expected value (80.96%) was known in advance; a pass/fail-only check would have shipped it green.Verification
Every value cross-checked against an independently written scorer, not the gate's own output.
Load-bearing negative controls — the gate must still fail when coverage genuinely drops:
Single-line discrimination at the floor — the gate is non-vacuous.
What this does NOT fix
The ~0.6 pp run-to-run jitter is untouched; only the level moves.
h2_proxynow sits ~1.0–1.6 pp above the floor instead of straddling it — a real margin, not a comfortable one. If it drifts down again the honest fix is real coverage (S43 option (c)), not another metric change.Follow-on
With this landed,
Coverage (per-module hot-path >= 80%)becomes safe to add to required status checks. Before it, requiring that check would have blocked ~half of all PRs on a measurement artifact. Owner copy-paste pack for branch protection + theSOAK_*values:audit/release/owner-actions.md.