Skip to content

ci: put the required-context entry criteria where a person will meet them - #1092

Merged
andrewgazelka merged 1 commit into
mainfrom
ci/flake-gate-entry-criteria
Jul 30, 2026
Merged

ci: put the required-context entry criteria where a person will meet them#1092
andrewgazelka merged 1 commit into
mainfrom
ci/flake-gate-entry-criteria

Conversation

@andrewgazelka

Copy link
Copy Markdown
Member

Follow-up to #1090.

The four measured criteria for making the differential verdict a required status check were only in the summary nix/ci/delta-gate.sh renders. That summary is read when the pipeline runs, and every trigger has been workflow_dispatch since #1088, so nobody meets them.

They move into the header of nix/ci/flake-gate.nix, which is the file anybody asking "why does this not block anything" opens first. Each criterion carries its measured value and why the number is what it is:

  1. Flake rate at or under 5%. Measured 0.611 over the 18 main runs of 2026-07-28/29. 5% is derived rather than picked: the merge queue's grouping_strategy: ALLGREEN with max_entries_to_build: 5 means a flake ejects a batch of up to five pull requests rather than costing one author a re-run.
  2. At least 30 runs in the instability record, from P(proven) = 1 - q^n - (1-q)^n. At n=18 an 11% flake is still 12% likely to be unproven.
  3. At most 1 in 20 pull request runs blocked by a verdict a re-run then clears. The only one that measures the whole system, and only takeable while the verdict reports rather than blocks.
  4. p95 under 40 minutes over runs where every check reached a verdict. Stated that way because the observed 31 to 41 minutes is over red runs only and no green run of this gate has ever been observed.

The summary keeps the one-line statement that the verdict blocks nothing, plus ENG-10827, and points at the header instead of restating the criteria. One copy to keep true.

No behaviour change: comments plus one summary paragraph. checks.<system>.delta-gate still passes 65 of 65 in the sandbox, checks.<system>.flake-gate builds, nix run .#fmt clean.

Refs ENG-11433, ENG-10827.

…them

The four measured criteria for making the differential verdict a required
status check were only in the run summary that nix/ci/delta-gate.sh renders.
That summary is read when the pipeline runs, and every trigger has been
workflow_dispatch since #1088, so in practice nobody meets them.

They move to the header of nix/ci/flake-gate.nix, which is the file anybody
asking "why does this not block anything" opens first. The summary keeps the
one-line statement that the verdict blocks nothing and points at the header
rather than restating the criteria, so there is one copy to keep true.

No behaviour change: comments and one summary paragraph.
@andrewgazelka
andrewgazelka enabled auto-merge July 30, 2026 03:45
@andrewgazelka
andrewgazelka merged commit ee1139e into main Jul 30, 2026
@andrewgazelka
andrewgazelka deleted the ci/flake-gate-entry-criteria branch July 30, 2026 03:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant