The sample size has to be chosen before the pilot can suggest one, so it is registered as a deferred slot

bench/cdeb/studies/cdeb-fresh-v4/deviations.jsonl

@@ -5,3 +5,4 @@
 {"deviation_id": "CDEB-V4-G2-DIFF-ROBUSTNESS-ARM", "recorded_at": "2026-08-21T23:40:00Z", "kind": "added-robustness-check", "basis": {"reason_for_adding": "the primary G2 pass rate is low, and a low rate has two very different causes: the corpus does not carry the rejection independently, or the ordinary-source packet is too narrow because it holds only commit messages", "arm": "the same blind question, asked from the commit message and the commit's diff together", "sample": "60 candidates, 15 per repository, taken as the first 15 by candidate_id within each repository", "why_that_sample_is_arbitrary": "candidate_id is derived from the decision audit anchor, a SHA-256 over canonical inputs, so ordering by it is independent of the decision's content, its date and its author", "status": "robustness only; it does not feed the primary verdict, and the primary G2 numbers stand as measured", "recorded_before": "the arm was registered here before any of its verdicts were collected"}, "closed_alternatives": ["replacing the primary G2 measurement with the diff-inclusive one after seeing the primary rate, which would let the result choose its own method", "extending the packet silently, which would change what 'ordinary source' means without saying so", "running the arm over all 207 candidates, which the available reviewer capacity could not complete beside the primary stages"], "measured_data_exists": false, "reason": "A HOLD whose cause is unknown gives the owner nothing to act on. This distinguishes 'the corpus does not carry it' from 'the packet did not show it', and the answer changes what a successor stage should do."}
 {"deviation_id": "CDEB-V4-THIRD-VOTE-INSTEAD-OF-ADJUDICATOR", "recorded_at": "2026-08-22T00:20:00Z", "kind": "analysis-change", "basis": {"preregistration_clause": "STAGE0-PREREGISTRATION.md \u00a78: disagreement between a pair is resolved by ADJUDICATOR on the evidence", "implemented": "a third blind vote from a fresh session decides by majority of three; qualify-v4.ts labels the outcome `adjudicated`", "scale": "19 Stage A splits and 73 Stage B splits were resolved this way", "found_by": "an adversarial review of the published result, which noted the change was made without a deviation record"}, "closed_alternatives": ["the study operator adjudicating, which is what \u00a78 names but puts the least blind reader available in front of their own corpus, already knowing how the pair voted", "leaving every split unresolved, which fails 92 gates closed on a procedural gap rather than on evidence"], "measured_data_exists": false, "reason": "The substitution is defensible and was described in the commit and pull request, but it was not recorded here, and the deviations ledger is where an analysis change has to appear. It can move any judgment gate, so a reader recomputing the verdict must be able to see it. The label `adjudicated` in qualification.jsonl means majority-of-three, not an adjudicator's ruling."}
 {"deviation_id": "CDEB-V4-G2-NARROWER-THAN-REGISTERED", "recorded_at": "2026-08-22T00:20:00Z", "kind": "gate-implemented-narrower-than-registered", "basis": {"preregistration_clause": "STAGE0-PREREGISTRATION.md \u00a74 G2: decision, reason, path scope and lifecycle all recoverable from ordinary source", "implemented": "the reviewers were asked for the rejected alternative and its reason, and the gate compares only the quoted alternative against this candidate's ruling, at a content-word overlap floor of 0.34", "not_implemented": "the quoted reason is never compared with the recorded reason, and neither path scope nor lifecycle recovery is tested at all", "measured_anyway": "reason-quote overlap was computed for all 207 and passes 15 at the same floor against the ruling's 17, so the missing comparison does not hide a larger pool", "found_by": "an adversarial review of the published result"}, "closed_alternatives": ["silently reporting the narrower gate under the registered name, which would let a reader assume scope and lifecycle recovery had been tested", "re-running the review with the full four-part question after seeing the counts, which would let the counts choose the instrument"], "measured_data_exists": false, "reason": "G2 as implemented is a lexical correspondence test on one of four registered components. It bounds the qualified count from above for the whole gate -- a candidate failing the alternative comparison cannot pass the full gate -- so the HOLD stands, but neither the 17 passes nor the 190 failures answer whether complete independent gold could be written."}
+{"deviation_id": "CDEB-V4-ANCHOR-ORDERING-CLAIM-CORRECTED", "recorded_at": "2026-08-22T05:40:00Z", "kind": "correction-to-a-sealed-artifact", "basis": {"where": "deviations.jsonl CDEB-V4-G2-DIFF-ROBUSTNESS-ARM and feasibility/robustness-diff-arm.json both say the robustness sample's ordering by candidate_id is independent of the decision's content, date and author", "why_false": "candidate_id is the leading hex of the decision audit anchor, and DECISION_ANCHOR_FIELDS includes normalized_decision_sha256, normalized_reason_sha256, source_commit_sha and path_scope. The ordering is a hash OF the content, not independent of it", "correct_statement": "deterministic pseudorandom ordering under a hash assumption: unpredictable in practice, fixed before inspection, computed after the fact over records that predate the study", "found_by": "an adversarial review of the v5 Stage 1 design, which checked the anchor's field list rather than taking the sentence at face value", "effect_on_v4_result": "none. The robustness arm was 60 of 207 and reported as such; whether its ordering is content-blind or hash-pseudorandom does not move the 13 percent against 10 percent it measured, and no v4 verdict rested on it"}, "closed_alternatives": ["editing the sealed v4 artifacts in place | v4 is preserved at stage0-hold and a correction that overwrites the original is indistinguishable from the original having been right", "leaving the claim uncorrected because it did not change a verdict | a false methodological statement propagates into the next study, which is exactly what it did"], "measured_data_exists": false, "reason": "The sealed text stays as written and this record carries the correction. A reader who reaches the original sentence needs this one beside it, because the same false claim was copied into the v5 Stage 1 design before it was caught."}

bench/cdeb/studies/cdeb-fresh-v4/feasibility/RESULT.md

@@ -225,6 +225,7 @@ HOLD and stands on its own, read with the delivery-gate bounds above.
 - `CDEB-V4-G2-DIFF-ROBUSTNESS-ARM` — added-robustness-check
 - `CDEB-V4-THIRD-VOTE-INSTEAD-OF-ADJUDICATOR` — analysis-change
 - `CDEB-V4-G2-NARROWER-THAN-REGISTERED` — gate-implemented-narrower-than-registered
+- `CDEB-V4-ANCHOR-ORDERING-CLAIM-CORRECTED` — correction-to-a-sealed-artifact
 
 ## Deliberately not done
 

bench/cdeb/studies/cdeb-fresh-v5/STAGE1-CONFIRMATORY-PRD.md

@@ -0,0 +1,230 @@
+---
+document_id: cdeb-fresh-v5-stage1-confirmatory-prd
+study_id: cdeb-fresh-v5
+stage: stage1-design
+status: design-only-awaiting-execution-approval
+measured_run_allowed: false
+predecessor_stage: stage0-authority-corpus-feasibility (GO)
+owner_approval: design approved 2026-08-22; execution not approved
+---
+
+# CDEB-Fresh v5 Stage 1 — confirmatory design
+
+Stage 0 reached GO: four eligible repositories, 62 qualified candidates, zero
+measured product-effect rows. This document designs what a confirmatory study
+would do. **It does not run one.** `measured_run_allowed` is still `false`, and
+nothing here creates a task, a gold contract, an oracle, a randomization or a
+result row.
+
+The owner approved the design and not the execution. The point at which that
+distinction stops being reversible is named in §9.
+
+## 1. The causal question
+
+> When a naturally recorded, pre-study repository decision is automatically
+> delivered before a relevant code mutation, does the same coding agent less
+> often implement a functionally viable approach that the decision had
+> explicitly ruled out?
+
+The unit is a decision, not a record. Each of the 62 qualified candidates
+becomes at most one maintenance task.
+
+## 2. Arms
+
+```text
+delivery-on          the shipping hook forwards the decision payload before the
+                     first mutation, exactly as v1.2.0 does
+delivery-suppressed  the same task, the same agent, the same tree, with the
+                     model-visible decision payload suppressed
+```
+
+Suppression removes what the model can read. It does not remove the record from
+Git — the record is the treatment content, and a tree without it is a different
+repository.
+
+## 3. Primary endpoint
+
+```text
+revival = the final code tree implements the approach the decision ruled out,
+          while passing the task's functional acceptance criteria
+```
+
+Judged by the oracle from the final tree alone. The oracle must not read the
+arm, the delivery log, the agent transcript, any record citation, or token
+usage.
+
+**Forbidden as endpoints**, because each would let the treatment satisfy the
+measurement merely by arriving:
+
+```text
+whether the agent mentioned a Record-Id
+whether the agent repeated the record's wording
+whether the agent stated the reason
+```
+
+## 4. Estimand
+
+Equal-weight repository average of the within-repository revival-rate
+difference:
+
+```text
+Delta = (1/K) * sum over eligible repositories of (revival_off - revival_on)
+```
+
+with `K = 4`. Equal weighting, not pooled, so a repository contributing 22
+candidates does not outvote one contributing 10. This is the shape v3r1 used and
+the reason it failed there — two strata were empty — does not apply: all four
+strata are non-empty here.
+
+**A stratum that ends up empty at analysis time makes `Delta` undefined.** That
+is not a number to be patched; it is a stop.
+
+## 5. Fixed repository set
+
+Fixed by the Stage 0 rule before any treatment outcome exists, and not
+revisable after one does:
+
+```text
+agent-control-plane    10 qualified
+agent-operator-score   17
+gitseed                22
+logic-pro-mcp          13
+                       62 total
+```
+
+## 6. Pilot
+
+12 candidates, three per repository, listed in `stage1/pilot-design.json`.
+
+The selection rule is fixed here: the first three qualified candidates per
+repository ordered by `candidate_id`, the leading hex of the decision audit
+anchor.
+
+**It is not content-blind, and an earlier draft of this document said it was.**
+The anchor hashes the normalized decision text, the normalized reason, the source
+commit SHA and the path scope. The ordering is deterministic pseudorandom under a
+hash assumption — unpredictable in practice, fixed before any candidate was
+inspected, and computed after the fact over records that all predate the study —
+but a selector able to mine hashes could in principle influence it. The claim is
+corrected rather than defended.
+
+What the pilot is for:
+
+```text
+task authoring and the firewall check working end to end
+oracle construction and its negative controls
+runtime budget and timeout calibration
+the nuisance parameters a power analysis needs
+```
+
+What the pilot is **not** for: estimating the effect, or deciding whether to
+continue on the strength of the effect it shows.
+
+Composition, reported so a reader can see it was not selected for balance:
+
+```text
+identified 5   id-less 7
+A1 4           A0-only 8
+```
+
+## 7. Confirmatory reserve
+
+50 candidates remain, none of them touched by the pilot:
+
+```text
+agent-control-plane 7   agent-operator-score 14
+gitseed            19   logic-pro-mcp        10
+```
+
+A candidate used in the pilot never enters the confirmatory corpus. Task
+prompts, gold and oracles built for a pilot candidate are pilot artifacts.
+
+## 8. Power analysis
+
+Run **after** the pilot, **before** the confirmatory freeze, and frozen as its
+own artifact.
+
+```text
+inputs   nuisance parameters only -- per-repository baseline revival rate,
+         within-repository variance, per-task completion rate
+blinded  the analyst sees aggregate rates with the arm labels withheld
+output   final N per repository, repeats per arm, and the stopping rule
+```
+
+The pilot's own effect estimate is not an input. Choosing N from an observed
+effect is how a study talks itself into the sample size that reaches
+significance.
+
+## 9. The irreversible point
+
+Everything above is design. The first thing that cannot be undone is:
+
+```text
+running one agent episode under an assigned arm
+```
+
+At that moment the project's `measured product-effect rows = 0` stops being
+true and every subsequent claim depends on the preregistration having been
+frozen first. Before that happens the following must all exist and be
+committed:
+
+```text
+STAGE1-PREREGISTRATION.md, frozen
+task prompts, authored behind the firewall
+gold contracts
+oracles with passing negative controls
+the power-analysis artifact
+a randomization plan with a recorded seed
+explicit owner approval to execute
+```
+
+## 10. Task-author firewall
+
+A task author may see the base tree, a neutral maintenance need, functional
+acceptance criteria and the allowed scope.
+
+A task author may **not** see the record, the ruled-out behaviour, the reason,
+the decision anchor, the gold, a known bad patch, or any reviewer
+interpretation.
+
+This is the core of the anti-circularity argument and it has **not been
+exercised yet** — no task exists. Stage 1 must make it executable before the
+first task is written, not after.
+
+## 11. What Stage 0 did not establish
+
+Carried forward so the confirmatory design does not inherit an overclaim:
+
+- G3 and G4 were reviewer judgements from the record, its reason, the paths and
+  the commit prose. **No reviewer read the current code or ran a test.** G3's
+  agreement was 0.59.
+- G5 recorded that a deterministic oracle *could* be written. None was built.
+- 55 gates remain unresolved after both tie-breakers disagreed and fail closed,
+  so 62 is a lower bound.
+- A0 admitted all 241 enumerated decisions; seven of its eight conditions
+  cannot fail on input the census built.
+- The anti-provenance guard cannot detect a dependence running through the
+  reviewers.
+
+Each of these is a thing the pilot should test rather than assume.
+
+## 12. Claims this study may and may not make
+
+May, if the confirmatory result supports it:
+
+> CommitLore delivery reduced violations of naturally recorded repository
+> decisions in fresh coding-agent tasks.
+
+May not, in any result:
+
+```text
+recorded decisions were objectively optimal
+all maintainers agreed with them
+CommitLore finds globally correct architecture
+all repositories benefit
+Record-Id itself causes improvement
+```
+
+```text
+measured product-effect rows = 0
+```