ci: judge pull requests on the delta from main, not on absolute green

[andrewgazelka]

The problem, with the evidence

On 2026-07-29 two correct pull requests each displayed exactly the failures the other one fixes.

• ci: fix three of the checks the Flake job was failing #1080 repairs completions-e2e and minecraft-literals. Its CI reported bedwars-dev-boot-e2e, smash-dev-boot-e2e, smash-e2e.
• fix(nix): bump index so a dev-profile C dependency configures again #1081 repairs the two dev-boot gates. Its CI reported completions-e2e, minecraft-literals, smash-e2e.

Neither could ever be green on its own, so neither could land, so the failures accumulated. Both were correct. Both looked broken. A gate that can only say "green" cannot say "better than before", and when main is already red that is the only question a reviewer has.

What this does

The gate compares this run's failing set against a baseline that main publishes on every push. A pull request that leaves the set no larger passes. One that adds a name blocks, and the summary names it.

Replayed against the real 1080/1081 sets, using this implementation:

scenario	verdict	detail
PR 1080	pass	fixed = [completions-e2e, minecraft-literals], three excused
PR 1081	pass	fixed = [bedwars-dev-boot-e2e, smash-dev-boot-e2e], three excused
breaks a check (derivation moved)	block	blocked = [proxy]
same derivation, unchanged check set	pass	reported nondeterministic, not blamed
same derivation, but the check set changed	block	blocked = [proxy]

Flakiness is answered with a proof, not a statistic

A retry cannot do this job. For a check failing 30% of the time, two failures in a row happen 9% of the time, so a retry can acquit and never convict. Any gate whose flake answer is a retry blocks honest work 9% of the time.

A derivation path is a Merkle root over every input derivation, so an unchanged path means an unchanged build closure. Therefore:

Same derivation, different outcome, is a proof of nondeterminism.

Not evidence, not a heuristic. If a check passed on the base and fails here on the byte-identical derivation, the diff did not cause it. Measured here:

run 30428441399 (34eb46c, 06:31Z)  differential-traces  ok
run 30429463435 (dd8311f, 06:49Z)  differential-traces  FAILED
both on /nix/store/fczpzka3v7c9npf4a3i730ahkncla3f5-check-hyperion-differential-traces.drv

java.lang.IllegalStateException: chunks were still not entity ticking after 600 ticks
        at VanillaTrace.tickServer(VanillaTrace.java:514)

A wall-clock timeout inside the sandbox. Nothing in the derivation moved; the runner was slower.

Applied at two scopes: within a verdict when the derivation is unchanged, and across a rolling record for the common case where a pull request moves the hash. The record keys on the derivation for the proof and promotes onto the attribute for the conclusion, because nondeterminism is a property of how a check is written and survives a change to its inputs.

What the 18 main runs of 2026-07-28/29 actually say

Every one of the 18 was red. Extracted from the Flake job logs; . = ok, X = FAILED, - = check did not exist yet, oldest leftmost.

smash-e2e                XXXXXXXXXXXXXXXXXX   18/18   deterministic
completions-e2e          XXXXXXXXXXXXXXXXXX   18/18   deterministic
smash-dev-boot-e2e       -----------XXXXXXX     7/7   deterministic
bedwars-dev-boot-e2e     -----------XXXXXXX     7/7   deterministic
minecraft-literals       ......XXXXXXXXXXXX   12/18   REGRESSION, see below
smash-hud-e2e            X....X.XX.X..X..XX    8/18   44% nondeterministic
differential-traces      .....XXXXX..X....X    7/18   39% nondeterministic
bedwars-bow-e2e          ........XX........    2/18   11% nondeterministic
smash-selector-e2e       ..................    0/18   clean

61% of runs hit at least one flake, 2.57 gate runs expected per clean pass. That is measured, not assumed, and the implementation reproduces it: dg flake-rate on the folded real history returns 0.611 over the same 18 runs.

Three things here were not previously known. differential-traces and bedwars-bow-e2e are flaky and nobody had noticed. smash-selector-e2e is no longer flaky. And minecraft-literals is not one of the standing five, it is a regression that landed mid-window and stuck while the gate was already red, which is consequence 2 of a permanently red gate caught in the act.

Why the identity has to be the derivation, demonstrated on that data

Folding the real 18-run history two ways:

A. every check given a constant derivation
   proven nondeterministic: bedwars-bow-e2e, differential-traces, minecraft-literals, smash-hud-e2e

B. modelling the one real derivation change verified from the logs
   (minecraft-literals 9md01i6l -> 3rwh12gk at run 30418671535)
   proven nondeterministic: bedwars-bow-e2e, differential-traces, smash-hud-e2e

A excuses a genuine regression as a coin flip. B does not. The derivation keying is the whole difference, on real data.

Guards, and each was watched failing

Immunity is withdrawn when the derivation does not describe the run. The general form, because the list will grow: a pull request forfeits immunity when it changes anything the derivation does not capture but the run depends on. Three known members: environment (.github/workflows/**, nixConfig), scheduling and concurrency (nix/ci/flake-gate.nix, e2e port offsets), and a changed check set (detected from the two documents rather than a path list, because adding a check adds contention under a concurrent gate and no path list would catch every way of doing that).

nix/ci/delta-gate-tests.sh is 65 cases with no network, no nix and no clock, wired as checks.<system>.delta-gate. It contains the inverse of every guard, because the failure mode of a suite like this is passing for the wrong reason. Six deliberate mutations of the verdict logic were each caught by a named case:

immunity ignores forfeiture          CAUGHT  forfeit: same-drv flip now BLOCKS
every drv counts as identical        CAUGHT  row2 new-failure: gate BLOCKS
cap ignores the shrink exemption     CAUGHT  cap: a PR that shrinks the set still lands
deleted check counts as repaired     CAUGHT  removed: not credited as a repair
eval failure no longer blocks        CAUGHT  eval failure: BLOCKS even though the base fails too
instability keyed on attr not drv    CAUGHT  fold: the check is proven unstable

What keeps the failing set from becoming permanent

• The ratchet, automatic and needing nobody: a pull request takes the newest baseline, so the instant a check passes on main it can never be excused again. Publishing on green is the revocation mechanism, not an optimisation.
• A cap of 6 in nix/ci/delta-gate.sh. Past it nothing lands except pull requests that shrink the set. It is a committed constant, editable downward only and deliberately never derived from an observed count: a suite with proven coin flips will eventually have a lucky night, and a self-lowering cap would latch onto it and freeze the repository the next ordinary day.
• A standing issue, one rewritten in place and closed automatically on green. In this repository rather than Linear, because the repository is public and an outside contributor meeting a red gate needs to read why without an internal account. Dry-run by default; GATE_ISSUE_APPLY: "1" is one line in the workflow.

The exclusion list is deleted

Same idea at lower fidelity, stored where only human attention could refresh it, and every fix was a merge conflict against every other fix in flight. The header of nix/ci/flake-gate.nix now says where "known broken" lives so nobody reinvents it. A check that genuinely cannot run in CI belongs out of checks in flake.nix, which is a truer home for "this cannot run here" than a CI-side exception.

This is NOT a required status context, and this PR does not make it one

gh api repos/hyperion-mc/hyperion/rulesets/566717 --jq '.rules[].type' returns seven rules and no required_status_checks; it is the only ruleset and branches/main/protection returns 404. So one approving review is the only thing between any pull request and main today, red pipeline or no pipeline. Making the verdict binding is one required_status_checks entry naming this job. Four criteria should hold first, none of which does:

1. Flake rate at or under 5% over 20 consecutive runs. Measured 61%.
2. At least 30 runs in the instability record. At n=18 an 11% flake like bedwars-bow-e2e is still 12% likely to be unproven; n=30 puts anything at or above 10% over 97% likely to be proven.
3. At most 1 in 20 pull request runs blocked by a verdict a re-run then clears, over 20 runs. Only measurable in this report-only phase, which is why it is its own step.
4. p95 wall time under 40 minutes. Measured 31 to 41 over the 18 runs, all red; no green run of this gate has ever been observed, so the green cost is unknown. The merge queue's check_response_timeout_minutes is 60 while this job is allowed 90, so those two must stop disagreeing before anything is required.

Criterion 4 makes #1084 (concurrent gate) a prerequisite. The gate is two long checks and a tail with a 4-second median, so the concurrency floor is about 11 minutes.

Also worth naming: the queue's grouping_strategy is ALLGREEN with max_entries_to_build: 5, so once checks are required one flake ejects a batch of five rather than costing one author a re-run.

Composing with #1087, which landed while this was being written

#1087 gave the gate a results.jsonl of {attr, outcome, drvPath} per check, and wrote the same rationale I had arrived at separately: "the same derivation with two different outcomes is proof the change did not cause the failure." So this rebases onto their file rather than replacing it. Their concurrent build, their drv_map, their realised() store oracle and their unquotable guard are all untouched. What this adds is excluded deleted, and a verdict stage at the end that reads results.jsonl and nothing else.

That seam is load-bearing and now demonstrated: the build loop changed from one nix build per name to one concurrent --keep-going build for the whole set, and no part of the verdict noticed.

I adopted their vocabulary (pass/fail, not ok/fail), and the library tests for fail and treats anything else as passing, so a third spelling arriving later reads as a pass rather than silently turning every check red.

This does NOT restore the CI triggers

#1088 and #1089 removed every automatic trigger while this was in flight. Three of the reasons given are measurements from this same investigation: 18 consecutive red runs, no required status context, and three checks poisoning 61% of runs so a correct change needed 2.6 attempts.

This change answers two of those three and restores none of the triggers. A differential verdict makes a red base survivable and stops the coin flips being blamed on anybody, but it does not make the checks green, and green-and-stays-green is the bar #1089 set. Restoring triggers is a decision for whoever set that bar, not a side effect of landing machinery that changes what the bar is worth.

The consequence, stated plainly: on workflow_dispatch alone this machinery is mostly inert. No push means no published baseline, so the first pull request to run under it will find none and be judged exactly as it is today. A manual dispatch on main does produce a baseline and does fold the first instability samples, and the publish conditions are written as github.ref == 'refs/heads/main' && github.event_name != 'pull_request' so they keep working unchanged whenever the triggers come back.

Verification

• nix build .#checks.aarch64-darwin.delta-gate passes in the sandbox: 65 of 65.
• nix build .#checks.aarch64-darwin.flake-gate builds, so shellcheck passes on the gate.
• nix run .#fmt makes no changes.
• Workflow YAML parses; job and permission structure checked with yq.
• Replayed against the real 1080/1081 sets and against the real 18-run history; dg flake-rate on the folded history returns 0.611, matching the hand measurement.

Not verified, and worth knowing before approving:

• No full nix run .#flake-gate on x86_64-linux. The verdict stage is exercised only through fixtures and replays.
• No live CI run of any of this. With triggers manual, this PR does not run its own CI, so the baseline fetch, the artifact publishing, the standing issue job and the permission set have never executed. The first workflow_dispatch on main after this merges is where they are first exercised. That is the largest untested surface here and I am not going to describe it as working.
• The standing issue writes nothing until GATE_ISSUE_APPLY is "1". Its body has been rendered locally against the real folded record; it has never posted.