Repository navigation
Add a Claude Code pilot of the response evaluation suite - #12
Merged
Merged
Conversation
Captures native Claude Code answers through the always-on hook for the twelve cases under no rules, 0.4.0, and 0.5.0, with blinded review. In this host 0.4.0 passed 12/12 and 0.5.0 passed 11/12 at 18.1% more characters, so the Codex-side gain did not reproduce. Recorded as evidence, not a gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Claude Code pilot of the response evaluation suite, so the 0.5.0 rules are measured on the host that ships them. The Codex pilot (#11) measured Codex answers; this one captures native Claude Code answers through the always-on hook.
Result: in Claude Code, 0.4.0 passed 12/12 and 0.5.0 passed 11/12, so the Codex-side gain did not reproduce here. 0.5.0 used 18.1% more characters than 0.4.0 and 11.2% fewer than no rules. The one 0.5.0 failure is a stated assumption ("The tests haven't been rerun yet") judged unsupported by the reviewer; no answer in any condition omitted a requested part or claimed unauthorized work. Details and limits are in
evals/claude-code/PILOT.md.scripts/collect_claude_answers.py: one freshclaude -psession per case,--setting-sources "" --tools "", rules loaded with--plugin-dirfrom a frozen checkout so the real hook injects them. Runs from an empty non-repository directory; a first capture run inside the checkout leaked branch and file names into a fictional scenario and was discarded before review.evals/claude-code/: manifest, frozen rule snapshots, 36 unedited answers, blinded review input and key, the reviewer's decisions, attached reviews, scored results, and the report.plugin-load-check.ymlreplays it too. Neither is a quality gate, because the candidate did not pass all cases and the report says so.README.mdlinks both pilots;evals/README.mddocuments the Claude Code collection step.The canonical rules are unchanged. Whether 0.5.0's extra length is worth it on Claude Code is left open; a shorter length rule tested on both hosts with repeated samples would be the next experiment.
Checked:
python -m unittest discover -s tests(48 tests, OK),python scripts/evaluate_responses.py evals/claude-code/pilot.json(replays),python scripts/build_codex.py --check(up to date),claude plugin validate .(passed),git diff --check(clean).🤖 Generated with Claude Code