The sample size has to be chosen before the pilot can suggest one, so it is registered as a deferred slot - #832
Merged
Merged
Conversation
… it is registered as a deferred slot Stage 0 reached GO and the owner approved the design and not the execution. This is the design: a confirmatory PRD, a preregistration, a pilot allocation and a power-analysis plan. No task, gold contract, oracle, arm assignment or agent episode exists, and `measured_run_allowed` is still false. The pilot is 12 candidates, three per repository, chosen by a rule fixed before any of them was looked at: the first three by `candidate_id`, which derives from the decision audit anchor and is therefore independent of the decision's content, date and author. It lands at 5 identified and 7 id-less, 4 corroborated and 8 not, which is reported so a reader can see it was not arranged. 50 candidates remain in reserve against a floor of 24, and a pilot candidate never enters the confirmatory corpus. The preregistration fixes everything the pilot cannot influence -- hypothesis and its direction, arms, endpoint, estimand, analysis, exclusion reasons, stopping rules -- and defers exactly four values to the power artifact: N per repository, repeats per arm, the stopping rule and the minimum detectable difference. Filling those in is the only amendment the document permits; anything else makes it a new preregistration. The power analysis is blinded to arm labels and may not read the pilot's effect estimate. The reason is narrow: a sample size chosen from an observed effect is a sample size chosen to reach significance. The estimand is the equal-weight repository average of the revival-rate difference, and an empty stratum makes it undefined. That is written as a stop, not as a number to patch -- v3r1 died of exactly that and recomputing over the surviving strata would have hidden it. Stage 0's limits are carried into both documents rather than left behind: no reviewer read code or ran a test, G3 agreement was 0.59, G5 recorded that an oracle could be written rather than building one, 55 gates fail closed so 62 is a lower bound, and the anti-provenance guard cannot see a dependence running through reviewers. Each is something the pilot should test rather than inherit. The irreversible point is named: running one agent episode under an assigned arm. Seven things must exist and be committed before it, and owner approval to execute is one of them. Record-Id: r-v5stage1design Provenance: authored Certainty: firm Blast: system Undo: easy Ruled-out: writing the final N now | it comes from nuisance parameters the pilot has not produced, and inventing it would make the power artifact a formality Ruled-out: letting the power analysis see the pilot's effect | choosing N from an observed effect chooses the N that reaches significance Ruled-out: selecting pilot candidates by hand for a balanced mix | the balance is an outcome of a content-blind rule here, and hand-picking would put the study operator's judgement inside the sample Ruled-out: creating pilot/ tasks/ gold/ oracles/ randomization/ directories to hold the design | those are the places a run writes to, and a test asserts none exists Ruled-out: recomputing the estimand over surviving strata if one empties | that is the failure v3r1 died of, and averaging over three strata while claiming four is a different study reported under the old name Limit: this registers a design. It does not show that a task can be authored behind the firewall, that an oracle can be built for any of the 62, or that a revival is detectable in a real tree -- the firewall has never been exercised because no task exists Limit: 62 is a Stage 0 lower bound under gates that fail closed on unresolved disagreement. If the pilot shows the oracle cannot be built for a class of these candidates, the reserve shrinks and the power analysis may return a required N the reserve cannot supply Verified: 33 tests pass across the four v5 files; both typechecks clean. The pilot allocation is recomputed from the qualification artifact inside the test rather than compared against a copy, the pilot and reserve are proved disjoint and to sum to 62, and no measured row for this study exists anywhere under bench/results CommitLore-Version: 2.0.0
CommitLore — record lintTrailers: clean — 2 commits in Active constraints for the paths this PR touchesLimits (10)
Ruled out (24)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
…efects are answered An adversarial review of the Stage 1 design, run before freezing, found six ways the study as written could let a wrong conclusion through. The preregistration now says so in its first paragraph and refuses to authorise a measured run. It is a draft, not a registration. The one that matters most is that the endpoint has never been shown judgeable. Stage 0 recorded that reviewers believed a deterministic oracle could be built; none was built, and determinism is not sensitivity. Each oracle needs a ruled-out patch that passes functional acceptance and trips the oracle, plus compliant patches that do not. Close behind is a selection freedom I did not see. The population is the 50 reserve candidates but N may be smaller, and nothing said which ones. A builder who finds an oracle easy to write for crisp boundaries and hard for awkward ones would shape the corpus by giving up. Every reserve candidate needs an outcome-blind buildability disposition before N is chosen. The interval I registered is invalid. Bootstrapping four fixed repositories treats them as drawn from a superpopulation, ignores variation inside each, and admits 256 distinct resamples however many are requested -- and I never stated a confidence level. I also deferred more than N: alpha, power, target effect and the stopping rule all have to be fixed before the pilot, because opaque arm labels hide direction and not magnitude. And the exclusions condition on post-treatment events, so a treatment that merely prevents task completion would score as preventing revival. Separately, one sentence I wrote was simply false. I said the pilot ordering by `candidate_id` is independent of the decision's content, date and author. The anchor hashes the normalized decision text, the normalized reason, the source commit SHA and the path scope -- it is a hash *of* the content. The ordering is deterministic pseudorandom under a hash assumption, fixed before any candidate was inspected, over records that all predate the study. That is a weaker and true claim. The same false sentence is in v4's robustness arm, which is sealed. It is not edited; a correction deviation is recorded beside it, and v4's generated result picks the new entry up because the drift check refused the stale copy. Record-Id: r-v5stage1failedreview Provenance: authored Certainty: firm Blast: system Undo: easy Ruled-out: patching the six defects into this document and calling it registered | its own section 7 says anything but the deferred N makes it a new preregistration, and quietly amending the document that defines what counts as amendment is the failure it exists to prevent Ruled-out: editing v4's sealed robustness artifacts to fix the claim | a correction that overwrites the original is indistinguishable from the original having been right Ruled-out: keeping the false ordering claim because it changed no verdict | it changed nothing in v4 and was copied straight into the v5 design, which is how a methodological error propagates Ruled-out: freezing anyway and treating the six as limitations | four of them are ways to reach a favourable answer that is not true, which is not a limitation Limit: two further review findings are recorded as limitations rather than defects -- the arms differ in payload volume and hook activity as well as decision content, and the record stays discoverable in Git in both arms, so the contrast is the total effect of automatic delivery rather than of the content or of access Limit: this commit records what is wrong with the design. It does not fix it. Several fixes are owner-level choices -- a co-primary task-success endpoint, a sham arm, an independent custodian for the pilot blind -- and none is mine to make Verified: 26 tests pass across the Stage 1 and v4 result files; both typechecks clean. The v4 drift check caught the stale result before this commit rather than after, which is the whole reason the report is generated CommitLore-Version: 2.0.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 0 reached GO and the owner approved the design, not the execution. This is the design: a confirmatory PRD, a preregistration, a pilot allocation and a power-analysis plan.
No task, gold contract, oracle, arm assignment or agent episode exists.
measured_run_allowedis stillfalse, and no measured row for this study exists anywhere underbench/results.Pilot — 12 candidates, chosen before anyone looked at them
Three per repository, by the first three
candidate_ids. That id derives from the decision audit anchor — a SHA-256 over canonical inputs — so the ordering is independent of the decision's content, date and author.It lands at 5 identified / 7 id-less, 4 corroborated / 8 not. Reported so a reader can see the balance is an outcome of a blind rule rather than an arrangement.
50 remain in reserve against a floor of 24, and a pilot candidate never enters the confirmatory corpus.
Preregistration — everything fixed except four values
Hypothesis and its direction, arms, endpoint, estimand, analysis, exclusion reasons and stopping rules are all fixed now. Deferred to the power artifact: N per repository, repeats per arm, the stopping rule, and the minimum detectable difference stated in advance.
Filling those in is the only amendment the document permits. Anything else makes it a new preregistration.
Power analysis is blind to the pilot's effect
It may read baseline rates, variance and completion rates. It may not read the estimated treatment effect, and the analyst sees arm labels withheld. A sample size chosen from an observed effect is a sample size chosen to reach significance.
An empty stratum is a stop, not a patch
The estimand is the equal-weight repository average of the revival-rate difference. If any repository contributes zero analysable tasks,
Deltais undefined and the study stops — it is not recomputed over the surviving strata. v3r1 died of exactly that, and recomputing would have hidden it.Stage 0's limits are carried forward, not left behind
No reviewer read code or ran a test; G3 agreement was 0.59; G5 recorded that an oracle could be written rather than building one; 55 gates fail closed so 62 is a lower bound; the anti-provenance guard cannot see a dependence running through reviewers. Each is something the pilot should test rather than inherit.
The irreversible point is named
Running one agent episode under an assigned arm. Seven things must exist and be committed before it — including explicit owner approval to execute.
Verification
33 tests pass across the four v5 files; both typechecks clean. The pilot allocation is recomputed from the qualification artifact inside the test rather than compared against a copy; the pilot and reserve are proved disjoint and to sum to 62; and no
tasks/,gold/,oracles/,pilot/,rows/orrandomization/directory exists.