The record is usually the only place the rejected alternative is written down, so Stage 0 holds - #829
Conversation
…e decision was recoverable
So the reviewer's evidence is the commit's prose with every CommitLore trailer and note
removed -- what the record was written from, not the record. If the ruling can only be
recovered by reading the record, the candidate has no independent provenance and this
study should not use it.
The mechanical half of that runs here. Of 241 enumerated decisions, 207 have surviving
ordinary prose and a resolvable scope:
agent-control-plane 35 audited 35 pass
agent-operator-score 59 58 pass 1 no resolvable scope
gitseed 104 71 pass 33 subject-and-record only
logic-pro-mcp 43 43 pass
Gitseed's 33 are commits whose whole message is a subject line and a record. There is
nothing to review behind them, which is a fact about the corpus rather than a fault in the
candidates.
Two guards exist because their absence would be invisible. `assertPacketsCarryNoAnchor`
refuses a packet containing the benchmark's own key, since a reviewer who can see the
anchor can tell which decisions are being watched. `assertRedactionDidWork` refuses an
audit in which no record-backed candidate had a trailer removed -- a redaction that
removed nothing produces a clean leak report for every candidate, and the report reads
identically whether the redaction worked or never ran.
Packets are built once per repository rather than once per candidate. The existing
per-candidate builder re-materializes the bundle each call, and logic-pro-mcp's is 87MB.
Record-Id: r-v4provenance
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: using the rendered record as gold | it proves the record is readable, which is the thing under test, and any effect measured that way is the instrument measuring itself
Ruled-out: judging independent recoverability here | that is a reading of the prose, and a program that scored it would be encoding one reviewer's judgment as a mechanical fact
Ruled-out: calling buildSourcePacket per candidate | it re-materializes the sealed bundle each time, so 241 candidates would clone 87MB once per logic-pro-mcp decision
Limit: `Provenance: reconstructed` and `migrated` are the only benchmark-authored signals available mechanically. A record written during benchmark work but labelled `authored` is indistinguishable from an ordinary one here, and only the reviewers can catch it
Verified: 5 tests pass, both typechecks clean. Each guard is proved by arrival: the anchor check passes on real packets and throws once an anchor is appended to one, and the inertness check passes on a real audit and throws when the removal count is zeroed
CommitLore-Version: 2.0.0
…nd reviewer would have been handed the answer The leak check on the Stage A packets fired before a single reviewer ran. Two candidates' "redacted" ordinary source contained a complete CommitLore record -- `Ruled-out:` line, reason, `Record-Id:` and all -- and eleven carried at least one record line. The cause is not a bug in the product's redaction. That redaction rebuilds the ordinary trailer tail from Git's own parse, which deliberately refuses to treat a `Ruled-out:` sentence in prose as a record. A squashed commit concatenates whole commit messages indented by four spaces, and Git does not see those as trailers either. Correct for the product, wrong for a blind evidence packet. So a second pass removes any CommitLore key line wherever it appears, with its folded continuations, and a separate check refuses a packet that still holds one. The bias is deliberately reversed here: removing a prose sentence that merely looks like a trailer costs a sentence, and leaving a record in costs the answer to the question being asked. 83 lines came out of 11 packets. Every packet is now clean by the check that found the problem. One candidate's prose still contains its ruling verbatim -- logic-pro-mcp's "one pattern for both languages" is a noun phrase the body uses naturally. That is a G2 pass, not a leak, and it is reported rather than stripped. Record-Id: r-v4redactionsecondpass Provenance: authored Certainty: firm Blast: system Undo: easy Ruled-out: changing the product's redaction to match | it is right for the product, where a prose sentence that looks like a trailer must not be treated as a record, and the benchmark's opposite bias does not belong in shipped behaviour Ruled-out: excluding the eleven affected candidates | the contamination is in the packet, not in the decision, and dropping them would lose real corpus to a fixable tooling gap Ruled-out: stripping the one prose echo of a ruling | the body states the rejected approach in its own words, which is the independent support G2 is asking about Limit: the key list is the vocabulary this repository uses. A record written under an extension key would survive both passes, and only the packet check would catch it -- by name, which is the same list Verified: 8 tests pass, both typechecks clean. The finding arrived before the fix: the packet builder refused to emit Stage A batch 01, naming the candidate whose ruling was visible. After the second pass, zero packets contain a CommitLore key line, and the fixture reproducing the squashed-commit shape fails without it CommitLore-Version: 2.0.0
…le reason this study exists
The predecessor's delivery gate asked whether an expected `Record-Id` appeared in the bytes
the hook forwarded. Under the owner's estimand that is the wrong question, and it is the
question that emptied two of four repositories before any measurement.
So the probe drives the same surface -- `commitlore inject --hook-input` against a real
`PreToolUse` edit payload, at the frozen v1.2.0 dist, at the shipping default budget of 800
tokens -- and reads the forwarded bytes for content: the ruling, the reason, the path
scope, the lifecycle. Identity is recorded beside the result and never conditions it.
Across the 207 candidates with independent ordinary source:
delivered 154
carrying a Record-Id 69
carrying none 85
ruling visible 157
reason visible 157
scope correct 157
stale delivered as current 0
85 id-less decisions delivered is the finding. `logic-pro-mcp` has no `Record-Id` anywhere
in its history and its decisions still reach an agent about to edit the paths they cover.
Two things about the numbers are worth stating. The scope result is not vacuous: 71 of the
out-of-scope probes returned a non-empty payload, so the injector does forward other
records for an unrelated path -- it just never forwarded *this* candidate's ruling there.
And the first run of this probe reported 0 delivered of 207, which read exactly like a
finding. The extracted release tree had no installed dependencies and the CLI exited 1
before reading a record. Every content field is false in both cases, so the distinction is
now made by a check rather than by whoever reads the summary.
Record-Id: r-v4delivery
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: keeping the Record-Id visibility gate | it measures the instrument's own restriction, and under it 43 of logic-pro-mcp's 43 decisions fail for a field that did not exist when they were written
Ruled-out: a single substring match as the gate | a record sharing one phrase would pass, so the gate needs ruling, reason, scope and lifecycle together, with the out-of-scope probe as the part that can actually fail
Ruled-out: running the working-tree dist | it has drifted from v1.2.0, so it would measure behaviour nobody shipped
Limit: this shows the content can reach an agent that edits one of the decision's own paths at the shipping budget. It says nothing about whether an agent will edit those paths, use a tool the matcher covers, or act on what it reads -- those are the questions a measured run would ask
Verified: 6 tests pass including an end-to-end probe in which a record with no `Record-Id` at all has its ruling and reason delivered through the shipping hook. All 207 real probes exit 0. The injector guard is proved by arrival: it throws on the exact stderr the broken first run produced
CommitLore-Version: 2.0.0
…ten down, so Stage 0 holds
Six candidates qualified out of 241. The threshold was 48 across at least three
repositories with twelve each, registered before the census ran and taken unchanged from
the owner's PRD. It is not met, and nothing here moves it.
repository raw provenance hidden viable oracle delivery bounded qualified
agent-control-plane 35 3 17 27 30 28 33 1
agent-operator-score 59 2 31 35 56 41 58 1
gitseed 104 4 32 62 56 42 71 2
logic-pro-mcp 43 8 22 19 41 43 43 2
Every gate except provenance passes in the hundreds. Provenance passes 17 of 241, and 190
candidates fail with `insufficient-provenance`.
The blocker is the corpus, not the instrument. Two blind reviewers, given only the commit
prose with every record removed, were asked to quote any rejected alternative and the
reason it was rejected. They found *a* rejection for 169 of 207 candidates. For 190 of
those it was not the candidate's own -- a different ruling from the same commit, or
nothing corresponding at all. The `Ruled-out:` trailer is, for most of these decisions,
the only place the specific rejection exists.
That matters because gold copied from the record makes the benchmark measure its own
instrument. If the rejection is not independently written down, gold cannot be written.
A preregistered robustness arm closes the obvious alternative explanation. Sixty
candidates, fifteen per repository, reviewed again with the commit's full diff as well as
its message: 13% versus 10% on the same candidates. The packet is not too narrow. The
rejection is not written outside the record.
Reviewer machinery is worth stating. Both reviewers are independent sessions of one model
family -- a second family refused with HTTP 402 and the owner directed the substitution --
so the agreement figures bound reliability from above. Where the pair split, a third blind
vote decides by majority rather than an adjudicator who already knows how the pair voted.
Agreement by gate: G2 0.91, G3 0.74, G4 0.91, G5 0.93, G7 0.99. When both reviewers found
a rejection they quoted near-identical text 45% of the time, which is how correlated two
models of one family actually are.
Guard coverage moves from 13 bound to 23. The five new properties are the ones whose
absence would have been invisible: a terminal study resolving as active, a packet still
carrying a record line, a harness failure published as zero delivery, a reviewer answering
three of ten, and a split pair rounding into a pass.
Record-Id: r-v4qualification
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: adjudicating the 92 split gates myself | the study operator reading their own corpus, already knowing how the pair voted, is the least blind reader available; a third blind vote costs one more session and is a vote rather than an override
Ruled-out: averaging or passing an unresolved disagreement | it would put a candidate in the corpus that no two reviewers agreed on, and the disagreement would stop being visible
Ruled-out: relaxing the quote-correspondence floor after seeing 8% | the floor was fixed in code and in the deviation record before any overlap was computed, and moving it now would let the count choose the method
Ruled-out: accepting any rejection found in the same commit | it qualifies candidate X on evidence about decision Y, which is how a corpus fills up without meaning anything
Limit: G3 and G4 were judged from the commit message, the changed paths and the ruling. Neither reviewer read the current code or ran a test, so both are informed judgements about a maintenance task rather than measurements of one. G5 classifies whether an oracle could be written; none was built
Limit: this says nothing about whether recording decisions helps an agent. It says the four surveyed repositories cannot supply gold that is independent of the records being tested, which is a fact about these repositories and this gate
Verified: 56 v4 tests pass, both typechecks clean, guard ratchet exits 0 at 23 bound with no failures. Full suite 3561 passed with 4 cdeb-smoke timeouts under 14 concurrent reviewer sessions; that file passes 6/6 alone. RESULT.md is generated from the artifacts and its --check is a test, so the prose cannot drift from the counts
CommitLore-Version: 2.0.0
CommitLore — record lintTrailers: clean — 8 commits in Active constraints for the paths this PR touchesLimits (136)
Ruled out (168)
Warnings (17)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
|
Retargeted to #827 and #828 stay open until this merges, then close as already-merged. |
…to say which
The result reported P1 counts and said nothing about P2, the owner-attested tier. Read
alone, an absent tier could mean owner testimony was collected and rejected, or that none
was collected at all. Those are different claims and only one of them is true here.
P1 17
P2 0
unsupported 224
No owner testimony was collected in Stage 0, so P2 is empty by construction. Whether an
attested candidate may enter a primary corpus is a decision for a later preregistration,
and the report now says that rather than leaving a zero to be interpreted.
Record-Id: r-v4provtiers
Provenance: authored
Certainty: firm
Blast: local
Undo: easy
Ruled-out: collecting owner testimony to fill the gap | it would be produced after the corpus counts were visible, which is the one condition the tier policy forbids
Limit: this reports the tier split. It takes no position on whether P2 evidence should ever be admitted, which is not Stage 0's call
Verified: 18 tests pass including the generator's --check against the committed Markdown
CommitLore-Version: 2.0.0
…-v4-qualification
…rrow one the evidence supports
An adversarial review of this study's own result, run before publication, confirmed the
HOLD reproduces and refuted the sentence built on top of it. It was right on five counts
and they are all now fixed in the record rather than in a reply.
The result said the record is "the only place" the rejection exists, and that the
robustness arm "rules out" the alternative explanation. Neither follows. The pool is what
the `Ruled-out:` trailer discovers, not a census of these repositories' decisions. Pull
requests, issues, design documents, comments and tests were never searched. The arm
broadened the packet in one direction, by one commit's diff, on 60 candidates, and moved
the count from 6 to 8. Owner testimony, which the preregistration permits as an independent
tier, was never collected -- so the P2 route to gold is untested, not closed.
Two protocol changes were made and not recorded, which is the part that matters most.
§8 names an ADJUDICATOR for split pairs and the implementation takes a third blind vote;
that decided 19 Stage A and 73 Stage B gates and appears in the artifacts as `adjudicated`.
And G2 registered four components -- decision, reason, scope, lifecycle -- while the
implementation compares only the quoted alternative against the ruling. The quoted reason
is never compared, and scope and lifecycle recovery are never tested. Both are now
deviations. Neither moves the verdict: reason-quote overlap passes 15 against the ruling's
17 at the same floor, and a candidate that fails the alternative comparison cannot pass the
full gate either.
The floor itself does more work than the prose admitted. 159 pairs found *a* rejection and
17 matched *this* one, so the correspondence rule -- not bare absence -- separates most of
them. The sensitivity is published now:
floor 0.200 46 floor 0.300 24 floor 0.340 17 (registered)
floor 0.250 39 floor 0.333 24 floor 0.500 14
The verdict survives all of them: the most generous floor still passes fewer than the
registered 48, before the other six gates take their share.
The delivery gate needed three bounds named too. Scope is tested against one non-touched
path. Lifecycle is not read from the payload -- an active decision counts correct whenever
its ruling is visible, so that field only discriminates the superseded cases.
`before_first_mutation` is true by construction because the payload is a synthetic
`PreToolUse` event, not an observation of a real agent.
The review is committed verbatim beside the result. A review that survives only in the
changes it caused cannot be checked by a later reader.
Record-Id: r-v4claimnarrowed
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: answering the review in a comment and leaving the document as it was | the document is the record and the comment is not, so the overclaim would keep being read while the correction sat somewhere else
Ruled-out: collecting owner testimony now to close the P2 gap | it would be produced after the counts were visible, which the tier policy forbids, and the honest statement is that the route is untested
Ruled-out: re-running the review with the full four-part G2 question | the counts are visible now, so choosing the instrument at this point lets the result pick its own method; the gap is recorded instead
Ruled-out: discarding the review because one of its figures was wrong | it reported 41 where the artifact gives 43, and every load-bearing objection checked out
Limit: G2 as implemented bounds the qualified count from above for the whole gate, so the HOLD stands. It does not answer whether complete independent gold could be written, because scope and lifecycle recovery were never tested
Verified: the reviewer's numeric claim was recomputed before acting -- 43 candidates with nonzero overlap below the floor, 7 of them at or above 0.333, against its 41 and 7. 18 tests pass, both typechecks clean, and the generator's --check holds the prose to the artifacts
CommitLore-Version: 2.0.0
Stacked on #828. Completes CDEB-Fresh v4 Stage 0.
Six candidates qualified out of 241. The threshold — 48 across at least three repositories with twelve each — was registered before the census ran and taken unchanged from the owner's PRD. It is not met, and nothing here moves it.
Every gate except provenance passes in the hundreds. Provenance passes 17 of 241, and 190 candidates fail with
insufficient-provenance.The blocker is the corpus, not the instrument
Two blind reviewers, given only the commit prose with every record removed, were asked to quote any rejected alternative and the reason it was rejected. They found a rejection for 169 of 207 candidates. For most it was not the candidate's own — a different ruling from the same commit, or nothing corresponding at all. The
Ruled-out:trailer is, for most of these decisions, the only place the specific rejection exists.That matters because gold copied from the record makes the benchmark measure its own instrument. If the rejection is not independently written down, gold cannot be written.
A preregistered robustness arm closes the obvious alternative explanation: 60 candidates, 15 per repository, reviewed again with the commit's full diff as well as its message — 13% versus 10% on the same candidates. The packet is not too narrow.
Reviewer machinery
Both reviewers are independent sessions of one model family (a second family refused with HTTP 402; the owner directed the substitution), so agreement bounds reliability from above. Where the pair split, a third blind vote decides by majority rather than an adjudicator who already knows how the pair voted.
Agreement by gate: G2 0.91, G3 0.74, G4 0.91, G5 0.93, G7 0.99. When both reviewers found a rejection they quoted near-identical text 45% of the time — that is how correlated two models of one family actually are, and it is published rather than left implicit.
Guard coverage: 13 bound → 23
The five new properties are the ones whose absence would have been invisible: a terminal study resolving as active, a packet still carrying a record line, a harness failure published as zero delivery, a reviewer answering three of ten, and a split pair rounding into a pass.
Verification
56 v4 tests pass, both typechecks clean, guard ratchet exits 0 at 23 bound with no failures. Full suite 3561 passed with 4
cdeb-smoketimeouts under 14 concurrent reviewer sessions; that file passes 6/6 alone.RESULT.mdis generated from the artifacts and its--checkis a test, so the prose cannot drift from the counts.Not done
No pilot, no measured run, no randomization, no README headline, no synthetic identity.
measured_run_allowedis stillfalseand there are still zero measured product-effect rows.