Repository navigation
fix(quality): count mutation timeouts as killed up to a cap - #233
Merged
Merged
Conversation
A timed-out mutant now counts as killed and stays listed. The gate prints a count line (and appends it to the job summary when GITHUB_STEP_SUMMARY is set) and fails when timeouts exceed 10% of the mutants, so CPU starvation cannot pass as kills. Survivors still fail. Both runners no longer fail their own action on a timeout, so the report reaches the gate. Generated-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:8786680e37c864fa;surface=sdk-cli
Generated-by: LLM effort=medium;harness=claude-code:2.1.325;model=anthropic:claude-sonnet-5-5;session=sha-256-64:8786680e37c864fa;surface=sdk-cli
Generated-by: LLM effort=medium;harness=claude-code:2.1.325;model=anthropic:claude-sonnet-5-5;session=sha-256-64:fd164b5936a0fa78;surface=sdk-cli
A run that is both errored and interrupted prints both verdicts in the not-cacheable message. With timeout no longer unstable, no other test listed two verdicts, so a mutant of the join separator survived. Generated-by: LLM effort=medium;harness=claude-code:2.1.325;model=anthropic:claude-sonnet-5-5;session=sha-256-64:ab53f8c879532bcb;surface=sdk-cli
Both runners now exit non-zero when timeouts exceed MUTATION_MAX_TIMEOUT_PERCENT, so Bazel does not cache a starved run. The gate reads the same constant, applies it to each report as well as the sum, rejects counts above 10^9, and requires exact per-report accounting with listed verdict lines matching the counts. Generated-by: LLM effort=high;harness=claude-code:2.1.325;model=anthropic:claude-sonnet-5-5;session=sha-256-64:29dbf9027bd7e66c;surface=sdk-cli
The count-bound check compares equal-length digit strings as text on purpose, and the per-report counts are set through eval, so shellcheck flagged SC2071, SC2034 and SC2154. Mark each with its reason and move the bound test out of the if condition so the directive can attach. Generated-by: LLM effort=high;harness=claude-code:2.1.325;model=anthropic:claude-sonnet-5-5;session=sha-256-64:29dbf9027bd7e66c;surface=sdk-cli
The gate now fails a report with a mutant.not_checked line even when the counts balance, and a mutant.<status> line whose status no verdict names. The mutmut runner records a failed `mutmut run` or `mutmut results` as interrupted, and runs the gate's accounting checks before returning success: counts that don't add up to the total, a mutant listed under a verdict its count denies, an unknown status, an unmapped mutant. It also bounds the timeout cap's length before converting it. The gomutants runner exits non-zero for a diff report whose selected mutants are not the total, or that lists an unmapped mutant, after writing the report. Bazel does not cache a failed action, so each of these runs is rerun instead of replayed. Generated-by: LLM effort=high;harness=claude-code:2.1.325;model=anthropic:claude-sonnet-5-5;session=sha-256-64:75b262e3e7e8f983;surface=sdk-cli
The report's consistency check reads a mutant line's status before the first equals sign. A mutant name can hold one, and mutmut's partition-to-rpartition change survived the self mutation gate. Test a known and an unknown status with an equals sign in the mutant name. Generated-by: LLM effort=high;harness=claude-code:2.1.325;model=anthropic:claude-sonnet-5-5;session=sha-256-64:75b262e3e7e8f983;surface=sdk-cli
Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:261946672f684d58;surface=sdk-cli
![tbhb-dev-agent-reviews[bot]](https://avatars.githubusercontent.com/u/288164213?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A starved CPU makes tests time out, and the strict gate fails the whole run on any timeout. One real run on 2 CPUs timed out 212 of 849 mutants. @tbhb decided that timeouts count as killed, with a cap so starvation can't pass as kills.
Behavior change
mutation_gate_testno longer fails on a timed-out mutant. It lists each one and printsmutation_gate: N timed out of M (X%), counted as killed, to$GITHUB_STEP_SUMMARYtoo when set. It fails when timeouts exceed 10% of the mutants, in the sum, and in each report. Exactly 10% passes. Survivors, uncovered mutants, and errored mutants still fail. The gate rejects counts past 10^9, counts that don't add up to the total, and listed verdicts that contradict the counts.The mutmut and gomutants runners write their report, then exit non-zero only past the cap. Within the cap they exit 0 and Bazel caches the pass, as the loosening decision means.
Decisions
Risk
A run with up to 10% timeouts now passes where it failed. The cap bounds that, and the report still lists every timeout.
Bazel caches a starved run within the cap as a pass and replays it later. It also caches full-scope runs when every report is empty, since the runner exits 0. Only the gate rejects that. It also enforces the 10^9 count cap. Runners skip that check, and they read
+5as 5.Verification
None beyond CI.
Opened-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli
Assisted-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli