Skip to content

Add CarMax task verifiers + judge rubrics#56

Closed
DEM1TASSE wants to merge 1 commit into
aiming-lab:mainfrom
DEM1TASSE:carmax-verify-only
Closed

Add CarMax task verifiers + judge rubrics#56
DEM1TASSE wants to merge 1 commit into
aiming-lab:mainfrom
DEM1TASSE:carmax-verify-only

Conversation

@DEM1TASSE

Copy link
Copy Markdown

Adds the per-task verifiers + judge rubrics for the CarMax mirror (the reviewer
deliverable). Submitted directly to main per the maintainer's request.

  • sites/carmax/verify/verify_<0..19>.py — one deterministic verifier per task
  • sites/carmax/verify/verify_lib.py — shared helpers, same interface/format as the merged
    sites/merriam_webster/verify/verify_lib.py (--run_dir/--initial_db/--after_db/ --container/--no_llm, Judge → JSON {task_id, pass, reason, evidence})
  • sites/carmax/tasks.jsonl — adds verifier_path + judge_rubric to each task

Each verifier is deterministic-first and checks three things: navigation (the agent
opened the relevant page — anti knowledge-shortcut), DB after-state for stateful tasks
(reservation / test-drive / saved / order / appraisal created or removed vs the seed), and
the answer against frozen ground truth (LLM-anchored only where phrasing is brittle,
always anchored on the ground truth).

Validated against the official react agent (agent_demo/agent.py): ran the agent per task,
then ran both graders (verifier + LLM judge) on each trajectory and compared. This caught and
fixed several over-strict verifier checks, and surfaced cases where the verifier is more
correct than a trajectory-only LLM judge (an empty final answer, and a reservation on the
wrong model year — both caught via DB after-state). Task #3 (Tesla Model 3 under 50k mi)
correctly fails as unsolvable.

The CarMax site itself is in PR #24 (still under review); these verifiers pair with it.

Per-task deterministic verifiers (verify_<N>.py) + shared verify_lib.py under
sites/carmax/verify/, plus verifier_path + judge_rubric on each task in tasks.jsonl.
Each verifier checks navigation (anti knowledge-shortcut), DB after-state for stateful
tasks, and the answer against frozen ground truth. Validated against the react agent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@DEM1TASSE DEM1TASSE changed the title Add CarMax task verifiers + judge rubrics (reviewer deliverable) Add CarMax task verifiers + judge rubrics Jul 3, 2026
@DEM1TASSE

Copy link
Copy Markdown
Author

Superseded by #57, which includes the CarMax site (preserving @Violet24K's authorship) together with the verifiers, so it's a complete mergeable contribution. Closing this verifier-only PR.

@DEM1TASSE DEM1TASSE closed this Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant