Skip to content

The sample size has to be chosen before the pilot can suggest one, so it is registered as a deferred slot - #832

Merged
MongLong0214 merged 2 commits into
mainfrom
cdeb-v5-stage1-design
Aug 22, 2026
Merged

The sample size has to be chosen before the pilot can suggest one, so it is registered as a deferred slot#832
MongLong0214 merged 2 commits into
mainfrom
cdeb-v5-stage1-design

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

Stage 0 reached GO and the owner approved the design, not the execution. This is the design: a confirmatory PRD, a preregistration, a pilot allocation and a power-analysis plan.

No task, gold contract, oracle, arm assignment or agent episode exists. measured_run_allowed is still false, and no measured row for this study exists anywhere under bench/results.

Pilot — 12 candidates, chosen before anyone looked at them

Three per repository, by the first three candidate_ids. That id derives from the decision audit anchor — a SHA-256 over canonical inputs — so the ordering is independent of the decision's content, date and author.

It lands at 5 identified / 7 id-less, 4 corroborated / 8 not. Reported so a reader can see the balance is an outcome of a blind rule rather than an arrangement.

50 remain in reserve against a floor of 24, and a pilot candidate never enters the confirmatory corpus.

Preregistration — everything fixed except four values

Hypothesis and its direction, arms, endpoint, estimand, analysis, exclusion reasons and stopping rules are all fixed now. Deferred to the power artifact: N per repository, repeats per arm, the stopping rule, and the minimum detectable difference stated in advance.

Filling those in is the only amendment the document permits. Anything else makes it a new preregistration.

Power analysis is blind to the pilot's effect

It may read baseline rates, variance and completion rates. It may not read the estimated treatment effect, and the analyst sees arm labels withheld. A sample size chosen from an observed effect is a sample size chosen to reach significance.

An empty stratum is a stop, not a patch

The estimand is the equal-weight repository average of the revival-rate difference. If any repository contributes zero analysable tasks, Delta is undefined and the study stops — it is not recomputed over the surviving strata. v3r1 died of exactly that, and recomputing would have hidden it.

Stage 0's limits are carried forward, not left behind

No reviewer read code or ran a test; G3 agreement was 0.59; G5 recorded that an oracle could be written rather than building one; 55 gates fail closed so 62 is a lower bound; the anti-provenance guard cannot see a dependence running through reviewers. Each is something the pilot should test rather than inherit.

The irreversible point is named

Running one agent episode under an assigned arm. Seven things must exist and be committed before it — including explicit owner approval to execute.

Verification

33 tests pass across the four v5 files; both typechecks clean. The pilot allocation is recomputed from the qualification artifact inside the test rather than compared against a copy; the pilot and reserve are proved disjoint and to sum to 62; and no tasks/, gold/, oracles/, pilot/, rows/ or randomization/ directory exists.

… it is registered as a deferred slot

Stage 0 reached GO and the owner approved the design and not the execution. This is the
design: a confirmatory PRD, a preregistration, a pilot allocation and a power-analysis
plan. No task, gold contract, oracle, arm assignment or agent episode exists, and
`measured_run_allowed` is still false.

The pilot is 12 candidates, three per repository, chosen by a rule fixed before any of them
was looked at: the first three by `candidate_id`, which derives from the decision audit
anchor and is therefore independent of the decision's content, date and author. It lands at
5 identified and 7 id-less, 4 corroborated and 8 not, which is reported so a reader can see
it was not arranged. 50 candidates remain in reserve against a floor of 24, and a pilot
candidate never enters the confirmatory corpus.

The preregistration fixes everything the pilot cannot influence -- hypothesis and its
direction, arms, endpoint, estimand, analysis, exclusion reasons, stopping rules -- and
defers exactly four values to the power artifact: N per repository, repeats per arm, the
stopping rule and the minimum detectable difference. Filling those in is the only amendment
the document permits; anything else makes it a new preregistration.

The power analysis is blinded to arm labels and may not read the pilot's effect estimate.
The reason is narrow: a sample size chosen from an observed effect is a sample size chosen
to reach significance.

The estimand is the equal-weight repository average of the revival-rate difference, and an
empty stratum makes it undefined. That is written as a stop, not as a number to patch --
v3r1 died of exactly that and recomputing over the surviving strata would have hidden it.

Stage 0's limits are carried into both documents rather than left behind: no reviewer read
code or ran a test, G3 agreement was 0.59, G5 recorded that an oracle could be written
rather than building one, 55 gates fail closed so 62 is a lower bound, and the
anti-provenance guard cannot see a dependence running through reviewers. Each is something
the pilot should test rather than inherit.

The irreversible point is named: running one agent episode under an assigned arm. Seven
things must exist and be committed before it, and owner approval to execute is one of them.

Record-Id: r-v5stage1design
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: writing the final N now | it comes from nuisance parameters the pilot has not produced, and inventing it would make the power artifact a formality
Ruled-out: letting the power analysis see the pilot's effect | choosing N from an observed effect chooses the N that reaches significance
Ruled-out: selecting pilot candidates by hand for a balanced mix | the balance is an outcome of a content-blind rule here, and hand-picking would put the study operator's judgement inside the sample
Ruled-out: creating pilot/ tasks/ gold/ oracles/ randomization/ directories to hold the design | those are the places a run writes to, and a test asserts none exists
Ruled-out: recomputing the estimand over surviving strata if one empties | that is the failure v3r1 died of, and averaging over three strata while claiming four is a different study reported under the old name
Limit: this registers a design. It does not show that a task can be authored behind the firewall, that an oracle can be built for any of the 62, or that a revival is detectable in a real tree -- the firewall has never been exercised because no task exists
Limit: 62 is a Stage 0 lower bound under gates that fail closed on unresolved disagreement. If the pilot shows the oracle cannot be built for a class of these candidates, the reserve shrinks and the power analysis may return a required N the reserve cannot supply
Verified: 33 tests pass across the four v5 files; both typechecks clean. The pilot allocation is recomputed from the qualification artifact inside the test rather than compared against a copy, the pilot and reserve are proved disjoint and to sum to 62, and no measured row for this study exists anywhere under bench/results
CommitLore-Version: 2.0.0
@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 2 commits in origin/main..d23aa23623113bae17c56a11c8a9f93c1aa919d2
Active constraints: 10 limits · 24 ruled-out · 0 warnings — from 7 records over 8 changed paths

Active constraints for the paths this PR touches

Limits (10)

  • r-v5stage1failedreview d23aa23 — two further review findings are recorded as limitations rather than defects -- the arms differ in payload volume and hook activity as well as decision content, and the record stays discoverable in Git in both arms, so the contrast is the total effect of automatic delivery rather than of the content or of access
  • r-v5stage1failedreview d23aa23 — this commit records what is wrong with the design. It does not fix it. Several fixes are owner-level choices -- a co-primary task-success endpoint, a sham arm, an independent custodian for the pilot blind -- and none is mine to make
  • r-v5stage1design 8eb65c2 — this registers a design. It does not show that a task can be authored behind the firewall, that an oracle can be built for any of the 62, or that a revival is detectable in a real tree -- the firewall has never been exercised because no task exists
  • r-v5stage1design 8eb65c2 — 62 is a Stage 0 lower bound under gates that fail closed on unresolved disagreement. If the pilot shows the oracle cannot be built for a class of these candidates, the reserve shrinks and the power analysis may return a required N the reserve cannot supply
  • r-v4claimnarrowed a74aaa5 — G2 as implemented bounds the qualified count from above for the whole gate, so the HOLD stands. It does not answer whether complete independent gold could be written, because scope and lifecycle recovery were never tested
  • r-v4provtiers e9b167a — this reports the tier split. It takes no position on whether P2 evidence should ever be admitted, which is not Stage 0's call
  • r-v4qualification b8ff1b9 — G3 and G4 were judged from the commit message, the changed paths and the ruling. Neither reviewer read the current code or ran a test, so both are informed judgements about a maintenance task rather than measurements of one. G5 classifies whether an oracle could be written; none was built
  • r-v4qualification b8ff1b9 — this says nothing about whether recording decisions helps an agent. It says the four surveyed repositories cannot supply gold that is independent of the records being tested, which is a fact about these repositories and this gate
  • r-v4delivery d2ff392 — this shows the content can reach an agent that edits one of the decision's own paths at the shipping budget. It says nothing about whether an agent will edit those paths, use a tool the matcher covers, or act on what it reads -- those are the questions a measured run would ask
  • r-v4stage0id 6b427af — this proves a terminal study cannot be resolved as active through the declaration. It does not stop a caller that names a study root directly, which is why the measured-run gate is separate and still shut

Ruled out (24)

  • r-v5stage1failedreview d23aa23 — patching the six defects into this document and calling it registered | its own section 7 says anything but the deferred N makes it a new preregistration, and quietly amending the document that defines what counts as amendment is the failure it exists to prevent
  • r-v5stage1failedreview d23aa23 — editing v4's sealed robustness artifacts to fix the claim | a correction that overwrites the original is indistinguishable from the original having been right
  • r-v5stage1failedreview d23aa23 — keeping the false ordering claim because it changed no verdict | it changed nothing in v4 and was copied straight into the v5 design, which is how a methodological error propagates
  • r-v5stage1failedreview d23aa23 — freezing anyway and treating the six as limitations | four of them are ways to reach a favourable answer that is not true, which is not a limitation
  • r-v5stage1design 8eb65c2 — writing the final N now | it comes from nuisance parameters the pilot has not produced, and inventing it would make the power artifact a formality
  • r-v5stage1design 8eb65c2 — letting the power analysis see the pilot's effect | choosing N from an observed effect chooses the N that reaches significance
  • r-v5stage1design 8eb65c2 — selecting pilot candidates by hand for a balanced mix | the balance is an outcome of a content-blind rule here, and hand-picking would put the study operator's judgement inside the sample
  • r-v5stage1design 8eb65c2 — creating pilot/ tasks/ gold/ oracles/ randomization/ directories to hold the design | those are the places a run writes to, and a test asserts none exists
  • r-v5stage1design 8eb65c2 — recomputing the estimand over surviving strata if one empties | that is the failure v3r1 died of, and averaging over three strata while claiming four is a different study reported under the old name
  • r-v4claimnarrowed a74aaa5 — answering the review in a comment and leaving the document as it was | the document is the record and the comment is not, so the overclaim would keep being read while the correction sat somewhere else
  • r-v4claimnarrowed a74aaa5 — collecting owner testimony now to close the P2 gap | it would be produced after the counts were visible, which the tier policy forbids, and the honest statement is that the route is untested
  • r-v4claimnarrowed a74aaa5 — re-running the review with the full four-part G2 question | the counts are visible now, so choosing the instrument at this point lets the result pick its own method; the gap is recorded instead
  • r-v4claimnarrowed a74aaa5 — discarding the review because one of its figures was wrong | it reported 41 where the artifact gives 43, and every load-bearing objection checked out
  • r-v4provtiers e9b167a — collecting owner testimony to fill the gap | it would be produced after the corpus counts were visible, which is the one condition the tier policy forbids
  • r-v4qualification b8ff1b9 — adjudicating the 92 split gates myself | the study operator reading their own corpus, already knowing how the pair voted, is the least blind reader available; a third blind vote costs one more session and is a vote rather than an override
  • r-v4qualification b8ff1b9 — averaging or passing an unresolved disagreement | it would put a candidate in the corpus that no two reviewers agreed on, and the disagreement would stop being visible
  • r-v4qualification b8ff1b9 — relaxing the quote-correspondence floor after seeing 8% | the floor was fixed in code and in the deviation record before any overlap was computed, and moving it now would let the count choose the method
  • r-v4qualification b8ff1b9 — accepting any rejection found in the same commit | it qualifies candidate X on evidence about decision Y, which is how a corpus fills up without meaning anything
  • r-v4delivery d2ff392 — keeping the Record-Id visibility gate | it measures the instrument's own restriction, and under it 43 of logic-pro-mcp's 43 decisions fail for a field that did not exist when they were written
  • r-v4delivery d2ff392 — a single substring match as the gate | a record sharing one phrase would pass, so the gate needs ruling, reason, scope and lifecycle together, with the out-of-scope probe as the part that can actually fail
  • r-v4delivery d2ff392 — running the working-tree dist | it has drifted from v1.2.0, so it would measure behaviour nobody shipped
  • r-v4stage0id 6b427af — keeping a hardcoded list of terminal study ids in the resolver | the list and the studies drift apart silently, and the drift shows up as a terminated study resolving cleanly
  • r-v4stage0id 6b427af — leaving the declaration at null and passing the study root explicitly everywhere | every caller then carries the choice, and the one caller that forgets picks a default nobody reviewed
  • r-v4stage0id 6b427af — reusing the v3r1 study directory under a new name | §4.3 requires a new study id, and a renamed directory keeps the qualification verdicts this estimand discards

git log --follow accepts exactly one pathspec, so renames are not followed for 8 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

…efects are answered

An adversarial review of the Stage 1 design, run before freezing, found six ways the study
as written could let a wrong conclusion through. The preregistration now says so in its
first paragraph and refuses to authorise a measured run. It is a draft, not a registration.

The one that matters most is that the endpoint has never been shown judgeable. Stage 0
recorded that reviewers believed a deterministic oracle could be built; none was built, and
determinism is not sensitivity. Each oracle needs a ruled-out patch that passes functional
acceptance and trips the oracle, plus compliant patches that do not.

Close behind is a selection freedom I did not see. The population is the 50 reserve
candidates but N may be smaller, and nothing said which ones. A builder who finds an oracle
easy to write for crisp boundaries and hard for awkward ones would shape the corpus by
giving up. Every reserve candidate needs an outcome-blind buildability disposition before
N is chosen.

The interval I registered is invalid. Bootstrapping four fixed repositories treats them as
drawn from a superpopulation, ignores variation inside each, and admits 256 distinct
resamples however many are requested -- and I never stated a confidence level. I also
deferred more than N: alpha, power, target effect and the stopping rule all have to be
fixed before the pilot, because opaque arm labels hide direction and not magnitude. And the
exclusions condition on post-treatment events, so a treatment that merely prevents task
completion would score as preventing revival.

Separately, one sentence I wrote was simply false. I said the pilot ordering by
`candidate_id` is independent of the decision's content, date and author. The anchor hashes
the normalized decision text, the normalized reason, the source commit SHA and the path
scope -- it is a hash *of* the content. The ordering is deterministic pseudorandom under a
hash assumption, fixed before any candidate was inspected, over records that all predate
the study. That is a weaker and true claim.

The same false sentence is in v4's robustness arm, which is sealed. It is not edited; a
correction deviation is recorded beside it, and v4's generated result picks the new entry up
because the drift check refused the stale copy.

Record-Id: r-v5stage1failedreview
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: patching the six defects into this document and calling it registered | its own section 7 says anything but the deferred N makes it a new preregistration, and quietly amending the document that defines what counts as amendment is the failure it exists to prevent
Ruled-out: editing v4's sealed robustness artifacts to fix the claim | a correction that overwrites the original is indistinguishable from the original having been right
Ruled-out: keeping the false ordering claim because it changed no verdict | it changed nothing in v4 and was copied straight into the v5 design, which is how a methodological error propagates
Ruled-out: freezing anyway and treating the six as limitations | four of them are ways to reach a favourable answer that is not true, which is not a limitation
Limit: two further review findings are recorded as limitations rather than defects -- the arms differ in payload volume and hook activity as well as decision content, and the record stays discoverable in Git in both arms, so the contrast is the total effect of automatic delivery rather than of the content or of access
Limit: this commit records what is wrong with the design. It does not fix it. Several fixes are owner-level choices -- a co-primary task-success endpoint, a sham arm, an independent custodian for the pilot blind -- and none is mine to make
Verified: 26 tests pass across the Stage 1 and v4 result files; both typechecks clean. The v4 drift check caught the stale result before this commit rather than after, which is the whole reason the report is generated
CommitLore-Version: 2.0.0
@MongLong0214
MongLong0214 merged commit 7d9721e into main Aug 22, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant