Skip to content

Add CarMax task verifiers + judge rubrics (reviewer deliverable)#1

Closed
DEM1TASSE wants to merge 1 commit into
Violet24K:mainfrom
DEM1TASSE:carmax-verifiers
Closed

Add CarMax task verifiers + judge rubrics (reviewer deliverable)#1
DEM1TASSE wants to merge 1 commit into
Violet24K:mainfrom
DEM1TASSE:carmax-verifiers

Conversation

@DEM1TASSE

@DEM1TASSE DEM1TASSE commented Jul 2, 2026

Copy link
Copy Markdown

Hi @Violet24K — this adds the per-task verifiers + judge rubrics for your CarMax
mirror. The WebHarbor organizer now asks the reviewer (not the task author) to provide
these for each task, so that task-writing and verifier-writing are done by different people
as a cross-check. I'm the reviewer for PR aiming-lab#24, so I wrote them.

Could you merge this into your CarMax PR branch (main)? Since PR aiming-lab#24's head is this
branch, merging here makes PR aiming-lab#24 automatically carry the verifiers upstream — and this way
both of us are credited as contributors (your site commits stay yours, the verifier
commit stays mine). Please use a regular merge, not squash, so commit authorship is
preserved.

What's in it (only the verifier layer — the site comes from your branch)

  • sites/carmax/verify/verify_<0..19>.py — one deterministic verifier per task
  • sites/carmax/verify/verify_lib.py — shared helpers, same interface/format as the merged
    sites/merriam_webster/verify/verify_lib.py
  • sites/carmax/tasks.jsonl — adds verifier_path + judge_rubric to each task

Each verifier checks navigation (agent opened the right page — anti knowledge-shortcut),
DB after-state for stateful tasks (reservation / test-drive / saved / order / appraisal
vs the seed), and the answer against frozen ground truth. Rubrics carry the same concrete
values so the LLM judge can grade too.

Validated against the official react agent (agent_demo/agent.py): verifier and LLM judge
agree except where the verifier is the more correct one (it caught an empty final answer and
a wrong-year reservation the LLM judge passed). Note: task aiming-lab#3 (Tesla Model 3 under 50k mi)
correctly fails as unsolvable, and a few site fixes are noted in the PR aiming-lab#24 review comments.

Per-task deterministic verifiers (verify_<N>.py) + shared verify_lib.py under
sites/carmax/verify/, plus verifier_path + judge_rubric on each task in tasks.jsonl.
Each verifier checks navigation (anti knowledge-shortcut), DB after-state for stateful
tasks, and the answer against frozen ground truth. Validated against the react agent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@DEM1TASSE DEM1TASSE changed the title Add CarMax task verifiers + judge rubrics Add CarMax task verifiers + judge rubrics (reviewer deliverable) Jul 2, 2026
@DEM1TASSE

Copy link
Copy Markdown
Author

Superseded by aiming-lab#56 — the maintainer asked to submit the verifiers directly to the main repo instead. Closing this companion PR.

@DEM1TASSE DEM1TASSE closed this Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant