Skip to content

Commit 0803929

Browse files
tannevaledclaude
andauthored
covratchet: ask the coverage question BEFORE the push, not 47 minutes later (#925)
The ratchet is correct and it caught the mistake. It caught it in CI, on a lane that runs the whole suite, about 47 minutes after the push -- twice in one day, on the SAME two functions, because a commit written to answer it did not move the measurement at all. A gate that refuses after the spend teaches the lesson. One that refuses before it prevents the mistake. `-require` answers the narrow question that is cheap locally: the functions this commit CLAIMS to have covered, are they actually at 100%? go test ./internal/vm/ -run FTP -coverprofile=/tmp/ftp.cov go tool cover -func=/tmp/ftp.cov > /tmp/ftp.func covratchet -profile /tmp/ftp.func -require ftpApplyOpts,ftpTimeoutErr A selective -run is sound here: a function at 100% under SOME tests is at 100% under all of them. It needs no record, so it runs on any profile. ⛔ THE REASON IT IS A COMMAND AND NOT A GREP, and the reason parse now returns the whole profile rather than only the sub-100 set: with the sub-100 set alone, "this function is fully covered" and "this profile has never heard of that name" are the SAME observation -- absence. A typo, a renamed function or a profile taken over the wrong package would every one of them read as success, and the check would be worse than useless because it would be believed. So a name that matches nothing FAILS, and says so: -- NOT IN THIS PROFILE -- the name matched nothing (1) -- ftpTimeoutErrr That is the `go test -run` failure mode exactly: exit 0 on a selection that matched nothing. Every verdict prints its denominator -- records matched out of a profile of N -- so a run that read nothing cannot pass for a clean one. Verified against the real profile that produced yesterday's red: the two functions report 100% and exit 0, while a single extra letter in one name exits 1 and names it. Also adds scripts/coverage/README.md, which did not exist: the ratchet was documented nowhere, and "N function(s) dropped below 100%" named no page to go to. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 1b8e177 commit 0803929

3 files changed

Lines changed: 268 additions & 18 deletions

File tree

‎scripts/coverage/README.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# The coverage ratchet
2+
3+
CI said **`N function(s) dropped below 100% -- named above`** and you landed
4+
here. This page says what that gate is, why it is not a percentage, and how to
5+
answer it without a round trip through CI.
6+
7+
## What is gated
8+
9+
Not the total. The **set of functions below 100%**, recorded per lane in
10+
`BELOW100.<lane>`, and that set may not grow.
11+
12+
A gate on the rounded total could not be calibrated. On `main` both POSIX lanes
13+
report `coverage: 100.0%` and pass *while printing* 19 (ubuntu) and 16 (macOS)
14+
functions below 100%, the lowest at 75.0%. The tolerance is whatever `%.1f`
15+
absorbs at the current repository size: nobody chose it, nothing states it, and
16+
it **shrinks as the codebase grows**. A function falling from 100% to 75% is
17+
free until it is suddenly fatal, and the commit that trips the gate is not the
18+
one that created the debt.
19+
20+
It was also arbitrary which lane you believed. On #727 the same commit gave
21+
99.9% on ubuntu (FAIL) and 100.0% on macOS (pass): ubuntu carries three
22+
filesystem-dependent functions macOS does not, so one further dead function
23+
tipped only one lane over the edge.
24+
25+
Percentages are still *reported*. They are not gated, because coverage is
26+
measured under `-race`, where a percentage jitters and a set does not — and a
27+
gate that flakes is a gate people learn to rerun.
28+
29+
## Answering it without waiting 47 minutes for CI
30+
31+
The lane that keeps the record runs the whole suite. Taking a full profile
32+
locally costs about the same, so the useful local question is narrower:
33+
34+
> the functions this commit **claims** to have covered — are they actually at
35+
> 100%?
36+
37+
```sh
38+
go test ./internal/vm/ -run 'FTP' -count=1 -coverprofile=/tmp/ftp.cov
39+
go tool cover -func=/tmp/ftp.cov > /tmp/ftp.func
40+
go run ./scripts/coverage/cmd/covratchet -profile /tmp/ftp.func \
41+
-require ftpApplyOpts,ftpTimeoutErr
42+
```
43+
44+
```
45+
required 2 function(s), matched 2 record(s) in a profile of 7347
46+
47+
all required functions at 100%
48+
```
49+
50+
A selective `-run` is fine here: a function reaching 100% under *some* tests is
51+
at 100% under all of them. `-require` takes bare names, or `file:name` when a
52+
name occurs in more than one file.
53+
54+
### ⛔ Why this is a command and not a `grep`
55+
56+
**A name that matches nothing fails.** With `grep`, a typo, a renamed function
57+
or a profile taken over the wrong package all print nothing — and nothing is
58+
exactly what success looks like:
59+
60+
```sh
61+
$ ... -require ftpApplyOpts,ftpTimeoutErrr # one extra 'r'
62+
required 2 function(s), matched 1 record(s) in a profile of 7347
63+
64+
-- NOT IN THIS PROFILE -- the name matched nothing (1) --
65+
ftpTimeoutErrr
66+
67+
::error::1 name(s) matched nothing; this profile cannot answer for them
68+
```
69+
70+
This is the same failure mode as `go test -run` exiting `ok` on a pattern that
71+
selected nothing. Every verdict therefore prints its **denominator** — how many
72+
records matched, out of how big a profile — so a run that read nothing cannot be
73+
mistaken for a clean one.
74+
75+
It exists because the ratchet, correct as it is, only refuses **after** the
76+
spend. Twice in one day a commit written to answer it did not move the
77+
measurement at all and came back red on the *same two functions*. A gate that
78+
refuses after the spend teaches the lesson; one that refuses before it prevents
79+
the mistake.
80+
81+
## When a function legitimately reaches 100%
82+
83+
The record is not updated by hand. On a green run of that lane:
84+
85+
```sh
86+
UPDATE_COVERAGE=1 ... # rewrites BELOW100.<lane>
87+
```
88+
89+
Leaving a fixed function in the record is not harmless: a stale allowance is
90+
where the next regression hides.
91+
92+
## Exit status
93+
94+
| code | meaning |
95+
| --- | --- |
96+
| 0 | no function newly below 100% (or, with `-require`, all of them at 100%) |
97+
| 1 | a regression, or a required name that matched nothing |
98+
| 2 | **refused to decide** — the profile or the record could not be read |
99+
100+
A profile with no `total:` line is refused rather than reported clean: a parser
101+
that swallows what it does not understand turns a format change into a pass.

‎scripts/coverage/cmd/covratchet/main.go‎

Lines changed: 109 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -88,11 +88,16 @@ type entry struct {
8888
}
8989

9090
// parse reads `go tool cover -func` output and returns every function below
91-
// 100%, keyed by identity. The total line is recognised and skipped rather than
92-
// silently mis-parsed: a parser that swallows what it does not understand turns
93-
// a format change into a clean result.
94-
func parse(r io.Reader, what, mod string) (map[fn]entry, error) {
91+
// 100%, keyed by identity, plus EVERY function it read. The total line is
92+
// recognised and skipped rather than silently mis-parsed: a parser that swallows
93+
// what it does not understand turns a format change into a clean result.
94+
//
95+
// The second map exists for -require, which has to tell "this function is at
96+
// 100%" apart from "this profile has never heard of that function". The first
97+
// map cannot: both are simply absent from it.
98+
func parse(r io.Reader, what, mod string) (map[fn]entry, map[fn]entry, error) {
9599
out := make(map[fn]entry)
100+
all := make(map[fn]entry)
96101
seen := make(map[string]int) // (file, name) -> how many seen so far, in file order
97102
sc := bufio.NewScanner(r)
98103
sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
@@ -104,41 +109,42 @@ func parse(r io.Reader, what, mod string) (map[fn]entry, error) {
104109
}
105110
f := strings.Fields(line)
106111
if len(f) < 2 || !strings.HasSuffix(f[len(f)-1], "%") {
107-
return nil, fmt.Errorf("%s line %d: not a cover -func record: %q", what, n, line)
112+
return nil, nil, fmt.Errorf("%s line %d: not a cover -func record: %q", what, n, line)
108113
}
109114
pct, err := strconv.ParseFloat(strings.TrimSuffix(f[len(f)-1], "%"), 64)
110115
if err != nil {
111-
return nil, fmt.Errorf("%s line %d: %q is not a percentage", what, n, f[len(f)-1])
116+
return nil, nil, fmt.Errorf("%s line %d: %q is not a percentage", what, n, f[len(f)-1])
112117
}
113118
if f[0] == "total:" {
114119
seenTotal = true
115120
continue
116121
}
117122
if len(f) < 3 {
118-
return nil, fmt.Errorf("%s line %d: expected file:line: name pct, got %q", what, n, line)
123+
return nil, nil, fmt.Errorf("%s line %d: expected file:line: name pct, got %q", what, n, line)
119124
}
120125
file, ln := splitPos(f[0], mod)
121126
// The ordinal is assigned BEFORE the 100% filter, so it does not shift
122127
// when a sibling of the same name crosses the line.
123128
nameKey := file + "\x00" + f[1]
124129
ord := seen[nameKey]
125130
seen[nameKey] = ord + 1
131+
k := fn{file: file, name: f[1], ord: ord}
132+
all[k] = entry{fn: k, pct: pct, line: ln}
126133
if pct >= 100 {
127134
continue
128135
}
129-
k := fn{file: file, name: f[1], ord: ord}
130136
if prev, dup := out[k]; dup {
131-
return nil, fmt.Errorf("%s line %d: %s appears twice (was %.1f%%)", what, n, k, prev.pct)
137+
return nil, nil, fmt.Errorf("%s line %d: %s appears twice (was %.1f%%)", what, n, k, prev.pct)
132138
}
133139
out[k] = entry{fn: k, pct: pct, line: ln}
134140
}
135141
if err := sc.Err(); err != nil {
136-
return nil, fmt.Errorf("reading %s: %w", what, err)
142+
return nil, nil, fmt.Errorf("reading %s: %w", what, err)
137143
}
138144
if !seenTotal {
139-
return nil, fmt.Errorf("%s has no total: line -- a profile this tool could not read must not be reported as clean", what)
145+
return nil, nil, fmt.Errorf("%s has no total: line -- a profile this tool could not read must not be reported as clean", what)
140146
}
141-
return out, nil
147+
return out, all, nil
142148
}
143149

144150
// splitPos splits "<module>/internal/vm/vm.go:1473:" into its repo-relative path
@@ -235,10 +241,11 @@ func run(args []string, stdin io.Reader, stdout io.Writer) error {
235241
lane := fs.String("lane", "", "lane name, for the record's header")
236242
mod := fs.String("module", "github.com/go-embedded-ruby/ruby/", "module path prefix to strip from each file path")
237243
update := fs.Bool("update", false, "rewrite the record from this run and exit 0")
244+
require := fs.String("require", "", "comma-separated functions (`name` or `file:name`) that must be at 100% in this profile")
238245
if err := fs.Parse(args); err != nil {
239246
return err
240247
}
241-
if *record == "" {
248+
if *record == "" && *require == "" {
242249
return fmt.Errorf("-record is required")
243250
}
244251

@@ -251,11 +258,15 @@ func run(args []string, stdin io.Reader, stdout io.Writer) error {
251258
defer f.Close()
252259
src = f
253260
}
254-
got, err := parse(src, "profile", *mod)
261+
got, all, err := parse(src, "profile", *mod)
255262
if err != nil {
256263
return err
257264
}
258265

266+
if *require != "" {
267+
return requireFull(stdout, all, *require)
268+
}
269+
259270
if *update {
260271
f, err := os.Create(*record)
261272
if err != nil {
@@ -318,6 +329,90 @@ func run(args []string, stdin io.Reader, stdout io.Writer) error {
318329
return nil
319330
}
320331

332+
// requireFull answers one question, before the push rather than after the CI
333+
// run: are the functions this commit CLAIMS to have covered actually at 100% in
334+
// a profile taken from this working tree?
335+
//
336+
// ⛔ WHY IT EXISTS. The ratchet already catches this, but it catches it in CI,
337+
// 47 minutes later, on a lane that has to run the whole suite. Twice in one day
338+
// a commit written to answer the ratchet did not move the measurement at all
339+
// and came back red on the SAME two functions. A gate that refuses after the
340+
// spend teaches the lesson; a gate that refuses before it prevents the mistake.
341+
//
342+
// ⛔ A SPEC THAT MATCHES NOTHING IS A FAILURE, NOT A PASS. This is the whole
343+
// reason the tool takes the full profile and not just the sub-100 set: with the
344+
// sub-100 set alone, "ftpTimeoutErr is at 100%" and "there is no function here
345+
// by that name" are the same observation -- absence. A typo, a renamed
346+
// function, a profile taken over the wrong package would all read as success,
347+
// and the check would be worse than nothing because it would be believed. The
348+
// same shape as `go test -run` exiting 0 on a pattern that selected nothing.
349+
func requireFull(w io.Writer, all map[fn]entry, spec string) error {
350+
type want struct{ file, name string }
351+
var wants []want
352+
for _, s := range strings.Split(spec, ",") {
353+
s = strings.TrimSpace(s)
354+
if s == "" {
355+
continue
356+
}
357+
if i := strings.LastIndexByte(s, ':'); i >= 0 {
358+
wants = append(wants, want{file: s[:i], name: s[i+1:]})
359+
continue
360+
}
361+
wants = append(wants, want{name: s})
362+
}
363+
if len(wants) == 0 {
364+
return fmt.Errorf("-require named no function")
365+
}
366+
367+
var missing, below []string
368+
matched := 0
369+
for _, w2 := range wants {
370+
var hits []entry
371+
for k, e := range all {
372+
if k.name != w2.name {
373+
continue
374+
}
375+
// A bare name matches in any file; file:name matches that path or
376+
// any suffix of it, so a repo-relative path and a bare basename
377+
// both work.
378+
if w2.file != "" && k.file != w2.file && !strings.HasSuffix(k.file, "/"+w2.file) {
379+
continue
380+
}
381+
hits = append(hits, e)
382+
}
383+
if len(hits) == 0 {
384+
missing = append(missing, w2.name)
385+
continue
386+
}
387+
matched += len(hits)
388+
for _, e := range hits {
389+
if e.pct < 100 {
390+
below = append(below, fmt.Sprintf("%s %.1f%%", e.fn, e.pct))
391+
}
392+
}
393+
}
394+
sort.Strings(missing)
395+
sort.Strings(below)
396+
397+
// Report the denominator beside the verdict: a check that says only "OK"
398+
// cannot be told from one that looked at nothing.
399+
fmt.Fprintf(w, "required %d function(s), matched %d record(s) in a profile of %d\n",
400+
len(wants), matched, len(all))
401+
section(w, "NOT IN THIS PROFILE -- the name matched nothing", missing)
402+
section(w, "BELOW 100%", below)
403+
404+
if len(missing) > 0 {
405+
fmt.Fprintf(w, "\n::error::%d name(s) matched nothing; this profile cannot answer for them\n", len(missing))
406+
return errRegression
407+
}
408+
if len(below) > 0 {
409+
fmt.Fprintf(w, "\n::error::%d required function(s) below 100%% -- named above\n", len(below))
410+
return errRegression
411+
}
412+
fmt.Fprintln(w, "\nall required functions at 100%")
413+
return nil
414+
}
415+
321416
func section(w io.Writer, title string, items []string) {
322417
if len(items) == 0 {
323418
return

‎scripts/coverage/cmd/covratchet/main_test.go‎

Lines changed: 58 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,60 @@ func judge(t *testing.T, profile string, extra ...string) (string, error) {
2929
return out.String(), err
3030
}
3131

32+
// ⛔ -require's whole point is the THIRD case below. A check built on the
33+
// sub-100 set alone cannot tell "this function is fully covered" from "this
34+
// profile has never heard of that name" -- both are absence -- so a typo, a
35+
// rename or a profile taken over the wrong package would read as success. That
36+
// is the `go test -run` failure mode: exit 0 on a selection that matched
37+
// nothing.
38+
func TestRequireTellsCoveredFromAbsent(t *testing.T) {
39+
for _, tc := range []struct {
40+
name string
41+
require string
42+
wantBad bool
43+
wantStr string
44+
}{
45+
{"a function at 100% passes", "baz", false, "all required functions at 100%"},
46+
{"a function below 100% fails", "foo", true, "BELOW 100%"},
47+
// The control this tool exists for.
48+
{"a name that matches nothing FAILS", "nosuchfunc", true, "matched nothing"},
49+
// One good name must not cover for one bad one: the verdict is per name.
50+
{"a real name beside an absent one still fails", "baz,nosuchfunc", true, "matched nothing"},
51+
{"file:name narrows to one file", "internal/vm/c.go:baz", false, "all required functions at 100%"},
52+
// Right name, wrong file: this is an absence, not a pass.
53+
{"file:name that points at the wrong file fails", "internal/vm/a.go:baz", true, "matched nothing"},
54+
} {
55+
t.Run(tc.name, func(t *testing.T) {
56+
var out strings.Builder
57+
err := run([]string{"-require", tc.require}, strings.NewReader(profile2), &out)
58+
if bad := errors.Is(err, errRegression); bad != tc.wantBad {
59+
t.Errorf("regression=%v, want %v (err=%v)\n%s", bad, tc.wantBad, err, out.String())
60+
}
61+
if !strings.Contains(out.String(), tc.wantStr) {
62+
t.Errorf("output does not mention %q:\n%s", tc.wantStr, out.String())
63+
}
64+
// ⛔ The denominator is printed beside every verdict, so a run that
65+
// read nothing cannot be mistaken for a clean one.
66+
if !strings.Contains(out.String(), "in a profile of 3") {
67+
t.Errorf("the verdict does not report what it read:\n%s", out.String())
68+
}
69+
})
70+
}
71+
}
72+
73+
// -require needs no record: it answers from the profile alone, which is what
74+
// makes it runnable before a push rather than only in the lane that keeps one.
75+
func TestRequireNeedsNoRecordButStillRefusesAnEmptyList(t *testing.T) {
76+
var out strings.Builder
77+
if err := run([]string{"-require", " , "}, strings.NewReader(profile2), &out); err == nil || errors.Is(err, errRegression) {
78+
t.Errorf("an empty -require must be a usage refusal, got %v", err)
79+
}
80+
out.Reset()
81+
if err := run(nil, strings.NewReader(profile2), &out); err == nil {
82+
t.Error("without -record and without -require it must refuse")
83+
}
84+
}
85+
3286
// TestCovRatchetVerdicts: the answer must depend on the input, and in
3387
// particular a function newly below 100% must fail EVEN WHEN the rounded total
3488
// still reads 100.0% -- which is the whole defect this replaces. On main both
@@ -158,7 +212,7 @@ func TestSameFunctionTwiceIsLegalNotMalformed(t *testing.T) {
158212
const prof = "github.com/go-embedded-ruby/ruby/a.go:1:\tf\t50.0%\n" +
159213
"github.com/go-embedded-ruby/ruby/a.go:9:\tf\t60.0%\n" +
160214
"total:\t(statements)\t100.0%\n"
161-
got, err := parse(strings.NewReader(prof), "profile", "github.com/go-embedded-ruby/ruby/")
215+
got, _, err := parse(strings.NewReader(prof), "profile", "github.com/go-embedded-ruby/ruby/")
162216
if err != nil {
163217
t.Fatalf("two same-named functions must parse, got %v", err)
164218
}
@@ -261,7 +315,7 @@ github.com/x/y/internal/vm/csv.go:280: ToS 100.0%
261315
github.com/x/y/internal/vm/io.go:10: displayStr 90.0%
262316
total: (statements) 99.9%
263317
`
264-
got, err := parse(strings.NewReader(prof), "profile", "github.com/x/y")
318+
got, _, err := parse(strings.NewReader(prof), "profile", "github.com/x/y")
265319
if err != nil {
266320
t.Fatalf("parse: %v", err)
267321
}
@@ -324,11 +378,11 @@ total: (statements) 99.9%
324378
github.com/x/y/a.go:120: ToS 50.0%
325379
total: (statements) 99.9%
326380
`
327-
b, err := parse(strings.NewReader(before), "before", "github.com/x/y")
381+
b, _, err := parse(strings.NewReader(before), "before", "github.com/x/y")
328382
if err != nil {
329383
t.Fatalf("before: %v", err)
330384
}
331-
a, err := parse(strings.NewReader(after), "after", "github.com/x/y")
385+
a, _, err := parse(strings.NewReader(after), "after", "github.com/x/y")
332386
if err != nil {
333387
t.Fatalf("after: %v", err)
334388
}

0 commit comments

Comments
 (0)