fix(rubric): make the 23/23 run reproducible instead of merely asserted - #49
Merged
Conversation
An independent review of the article about this work found the disqualifying
defect, and it was mine: the first rubric — the one that scored 23/23 — was never
committed. `git log --all -- docs/rubrics/` returned exactly one commit. The only
trace of 23/23 anywhere was PROSE written by the person claiming it: a commit
message, a YAML comment, a report card.
That is precisely the category this rubric exists to reject. Evidence is observed,
never asserted — and the headline number of the whole exercise was asserted.
So it is now observable:
docs/rubrics/archive/agent-loop-engineering.v1.yml 26 items (23 verifiable + 3 gaps)
docs/rubrics/archive/AUDIT.v1.md its output: 23/23, gate PASS
python3 scripts/audit_loop_rubric.py \
--rubric docs/rubrics/archive/agent-loop-engineering.v1.yml
The file is labelled a RECONSTRUCTION, not a recovered original: it is the current
rubric minus the five second-pass items, committed after the fact. Saying so is the
difference between evidence and a better-dressed assertion.
`--rubric` is added to the checker for this, and the report now names which rubric
it scored, so an archived run cannot be mistaken for the live one.
Also fixes a dead pointer the same review caught: both files cited
`docs/skill/playbooks/operationalizing-a-paper-rubric-checklist-standa.md`, a path
that does not exist in this repo (the playbook ships with the anyagent skill). The
rules are now restated in place so each file stands alone.
606 tests pass, ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An independent review of the article about this work found the disqualifying defect, and it was mine.
The rubric that scored 23/23 was never committed.
git log --all -- docs/rubrics/returned exactly one commit. The only trace of 23/23 anywhere was prose written by the person claiming it — a commit message, a YAML comment, a report card.That is exactly the category this rubric exists to reject: evidence is observed, never asserted. The headline number of the whole exercise was asserted.
Now it is observable:
python3 scripts/audit_loop_rubric.py --rubric docs/rubrics/archive/agent-loop-engineering.v1.yml # **Conformance: 23/23 (100%)** ... gate **PASS**docs/rubrics/archive/agent-loop-engineering.v1.yml— 26 items (23 verifiable + 3 declared gaps)docs/rubrics/archive/AUDIT.v1.md— its outputThe file is labelled a reconstruction, not a recovered original: it is the current rubric minus the five second-pass items, committed after the fact. Saying so is the difference between evidence and a better-dressed assertion.
--rubricis added to the checker, and the report now names which rubric it scored so an archived run cannot be mistaken for the live one.Also fixes a dead pointer the same review caught: both files cited a
docs/skill/playbooks/...path that does not exist in this repo. The rules are restated in place.606 tests pass, ruff clean.
🤖 Generated with Claude Code