Repository navigation
covratchet: ask the coverage question before the push, not 47 minutes later - #925
Open
tannevaled wants to merge 1 commit into
Open
tannevaled wants to merge 1 commit into
tannevaled wants to merge 1 commit into
Conversation
… later
The ratchet is correct and it caught the mistake. It caught it in CI, on a
lane that runs the whole suite, about 47 minutes after the push -- twice in
one day, on the SAME two functions, because a commit written to answer it
did not move the measurement at all. A gate that refuses after the spend
teaches the lesson. One that refuses before it prevents the mistake.
`-require` answers the narrow question that is cheap locally: the functions
this commit CLAIMS to have covered, are they actually at 100%?
go test ./internal/vm/ -run FTP -coverprofile=/tmp/ftp.cov
go tool cover -func=/tmp/ftp.cov > /tmp/ftp.func
covratchet -profile /tmp/ftp.func -require ftpApplyOpts,ftpTimeoutErr
A selective -run is sound here: a function at 100% under SOME tests is at
100% under all of them. It needs no record, so it runs on any profile.
⛔ THE REASON IT IS A COMMAND AND NOT A GREP, and the reason parse now
returns the whole profile rather than only the sub-100 set: with the sub-100
set alone, "this function is fully covered" and "this profile has never
heard of that name" are the SAME observation -- absence. A typo, a renamed
function or a profile taken over the wrong package would every one of them
read as success, and the check would be worse than useless because it would
be believed. So a name that matches nothing FAILS, and says so:
-- NOT IN THIS PROFILE -- the name matched nothing (1) --
ftpTimeoutErrr
That is the `go test -run` failure mode exactly: exit 0 on a selection that
matched nothing. Every verdict prints its denominator -- records matched out
of a profile of N -- so a run that read nothing cannot pass for a clean one.
Verified against the real profile that produced yesterday's red: the two
functions report 100% and exit 0, while a single extra letter in one name
exits 1 and names it.
Also adds scripts/coverage/README.md, which did not exist: the ratchet was
documented nowhere, and "N function(s) dropped below 100%" named no page to
go to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
covratchet: ask the coverage question before the push, not 47 minutes later
Why
The coverage ratchet is correct, and it caught the mistake. It caught it in
CI, on a lane that runs the whole suite, about 47 minutes after the push —
twice in one day, on the same two functions, because a commit written to
answer it did not move the measurement at all.
A gate that refuses after the spend teaches the lesson. One that refuses
before it prevents the mistake.
What
-requiredoesThe narrow question that is cheap locally: the functions this commit claims
to have covered — are they actually at 100%?
A selective
-runis sound here: a function at 100% under some tests is at100% under all of them. It needs no record, so it runs on any profile.
⛔ Why it is a command and not a
grepThis is also why
parsenow returns the whole profile and not only the sub-100set. With the sub-100 set alone, "this function is fully covered" and "this
profile has never heard of that name" are the same observation — absence. A
typo, a renamed function, or a profile taken over the wrong package would every
one of them read as success, and the check would be worse than useless,
because it would be believed.
So a name that matches nothing fails, and says which:
That is the
go test -runfailure mode exactly: exit 0 on a selection thatmatched nothing. Every verdict prints its denominator — records matched out
of a profile of N — so a run that read nothing cannot pass for a clean one.
Verified
Against the real profile that produced yesterday's red, not a fixture: the two
functions report 100% and exit 0, and a single extra letter in one name
exits 1 and names it. Plus eight new unit cases covering both halves, including
file:namepointing at the wrong file — a right name in the wrong place is anabsence, not a pass.
Also
scripts/coverage/README.md, which did not exist. The ratchet was documentednowhere in the repository, so
N function(s) dropped below 100%named no pageto go to; now it explains what is gated (the set, not the total, and why a
percentage could not be calibrated), how to answer it locally, and what each
exit status means.
🤖 Generated with Claude Code