Add CarMax task verifiers + judge rubrics#56
Closed
DEM1TASSE wants to merge 1 commit into
Closed
Conversation
Per-task deterministic verifiers (verify_<N>.py) + shared verify_lib.py under sites/carmax/verify/, plus verifier_path + judge_rubric on each task in tasks.jsonl. Each verifier checks navigation (anti knowledge-shortcut), DB after-state for stateful tasks, and the answer against frozen ground truth. Validated against the react agent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
|
Superseded by #57, which includes the CarMax site (preserving @Violet24K's authorship) together with the verifiers, so it's a complete mergeable contribution. Closing this verifier-only PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the per-task verifiers + judge rubrics for the CarMax mirror (the reviewer
deliverable). Submitted directly to
mainper the maintainer's request.sites/carmax/verify/verify_<0..19>.py— one deterministic verifier per tasksites/carmax/verify/verify_lib.py— shared helpers, same interface/format as the mergedsites/merriam_webster/verify/verify_lib.py(--run_dir/--initial_db/--after_db/ --container/--no_llm,Judge→ JSON{task_id, pass, reason, evidence})sites/carmax/tasks.jsonl— addsverifier_path+judge_rubricto each taskEach verifier is deterministic-first and checks three things: navigation (the agent
opened the relevant page — anti knowledge-shortcut), DB after-state for stateful tasks
(reservation / test-drive / saved / order / appraisal created or removed vs the seed), and
the answer against frozen ground truth (LLM-anchored only where phrasing is brittle,
always anchored on the ground truth).
Validated against the official react agent (
agent_demo/agent.py): ran the agent per task,then ran both graders (verifier + LLM judge) on each trajectory and compared. This caught and
fixed several over-strict verifier checks, and surfaced cases where the verifier is more
correct than a trajectory-only LLM judge (an empty final answer, and a reservation on the
wrong model year — both caught via DB after-state). Task #3 (Tesla Model 3 under 50k mi)
correctly fails as unsolvable.
The CarMax site itself is in PR #24 (still under review); these verifiers pair with it.