Skip to content

covratchet: ask the coverage question before the push, not 47 minutes later - #925

Open
tannevaled wants to merge 1 commit into
mainfrom
covratchet-require
Open

tannevaled wants to merge 1 commit into
mainfrom
covratchet-require

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

covratchet: ask the coverage question before the push, not 47 minutes later

Why

The coverage ratchet is correct, and it caught the mistake. It caught it in
CI
, on a lane that runs the whole suite, about 47 minutes after the push —
twice in one day, on the same two functions, because a commit written to
answer it did not move the measurement at all.

A gate that refuses after the spend teaches the lesson. One that refuses
before it prevents the mistake.

What -require does

The narrow question that is cheap locally: the functions this commit claims
to have covered — are they actually at 100%?

go test ./internal/vm/ -run FTP -coverprofile=/tmp/ftp.cov
go tool cover -func=/tmp/ftp.cov > /tmp/ftp.func
covratchet -profile /tmp/ftp.func -require ftpApplyOpts,ftpTimeoutErr
required 2 function(s), matched 2 record(s) in a profile of 7347

all required functions at 100%

A selective -run is sound here: a function at 100% under some tests is at
100% under all of them. It needs no record, so it runs on any profile.

⛔ Why it is a command and not a grep

This is also why parse now returns the whole profile and not only the sub-100
set. With the sub-100 set alone, "this function is fully covered" and "this
profile has never heard of that name" are the same observation — absence.
A
typo, a renamed function, or a profile taken over the wrong package would every
one of them read as success, and the check would be worse than useless,
because it would be believed.

So a name that matches nothing fails, and says which:

$ covratchet -profile /tmp/ftp.func -require ftpApplyOpts,ftpTimeoutErrr
required 2 function(s), matched 1 record(s) in a profile of 7347

-- NOT IN THIS PROFILE -- the name matched nothing (1) --
   ftpTimeoutErrr

::error::1 name(s) matched nothing; this profile cannot answer for them

That is the go test -run failure mode exactly: exit 0 on a selection that
matched nothing. Every verdict prints its denominator — records matched out
of a profile of N — so a run that read nothing cannot pass for a clean one.

Verified

Against the real profile that produced yesterday's red, not a fixture: the two
functions report 100% and exit 0, and a single extra letter in one name
exits 1 and names it. Plus eight new unit cases covering both halves, including
file:name pointing at the wrong file — a right name in the wrong place is an
absence, not a pass.

Also

scripts/coverage/README.md, which did not exist. The ratchet was documented
nowhere in the repository, so N function(s) dropped below 100% named no page
to go to; now it explains what is gated (the set, not the total, and why a
percentage could not be calibrated), how to answer it locally, and what each
exit status means.

🤖 Generated with Claude Code

… later

The ratchet is correct and it caught the mistake. It caught it in CI, on a
lane that runs the whole suite, about 47 minutes after the push -- twice in
one day, on the SAME two functions, because a commit written to answer it
did not move the measurement at all. A gate that refuses after the spend
teaches the lesson. One that refuses before it prevents the mistake.

`-require` answers the narrow question that is cheap locally: the functions
this commit CLAIMS to have covered, are they actually at 100%?

    go test ./internal/vm/ -run FTP -coverprofile=/tmp/ftp.cov
    go tool cover -func=/tmp/ftp.cov > /tmp/ftp.func
    covratchet -profile /tmp/ftp.func -require ftpApplyOpts,ftpTimeoutErr

A selective -run is sound here: a function at 100% under SOME tests is at
100% under all of them. It needs no record, so it runs on any profile.

⛔ THE REASON IT IS A COMMAND AND NOT A GREP, and the reason parse now
returns the whole profile rather than only the sub-100 set: with the sub-100
set alone, "this function is fully covered" and "this profile has never
heard of that name" are the SAME observation -- absence. A typo, a renamed
function or a profile taken over the wrong package would every one of them
read as success, and the check would be worse than useless because it would
be believed. So a name that matches nothing FAILS, and says so:

    -- NOT IN THIS PROFILE -- the name matched nothing (1) --
       ftpTimeoutErrr

That is the `go test -run` failure mode exactly: exit 0 on a selection that
matched nothing. Every verdict prints its denominator -- records matched out
of a profile of N -- so a run that read nothing cannot pass for a clean one.

Verified against the real profile that produced yesterday's red: the two
functions report 100% and exit 0, while a single extra letter in one name
exits 1 and names it.

Also adds scripts/coverage/README.md, which did not exist: the ratchet was
documented nowhere, and "N function(s) dropped below 100%" named no page to
go to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant