Add CarMax task verifiers + judge rubrics (reviewer deliverable)#1
Closed
DEM1TASSE wants to merge 1 commit into
Closed
Add CarMax task verifiers + judge rubrics (reviewer deliverable)#1DEM1TASSE wants to merge 1 commit into
DEM1TASSE wants to merge 1 commit into
Conversation
Per-task deterministic verifiers (verify_<N>.py) + shared verify_lib.py under sites/carmax/verify/, plus verifier_path + judge_rubric on each task in tasks.jsonl. Each verifier checks navigation (anti knowledge-shortcut), DB after-state for stateful tasks, and the answer against frozen ground truth. Validated against the react agent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9 tasks
Author
|
Superseded by aiming-lab#56 — the maintainer asked to submit the verifiers directly to the main repo instead. Closing this companion PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi @Violet24K — this adds the per-task verifiers + judge rubrics for your CarMax
mirror. The WebHarbor organizer now asks the reviewer (not the task author) to provide
these for each task, so that task-writing and verifier-writing are done by different people
as a cross-check. I'm the reviewer for PR aiming-lab#24, so I wrote them.
Could you merge this into your CarMax PR branch (
main)? Since PR aiming-lab#24's head is thisbranch, merging here makes PR aiming-lab#24 automatically carry the verifiers upstream — and this way
both of us are credited as contributors (your site commits stay yours, the verifier
commit stays mine). Please use a regular merge, not squash, so commit authorship is
preserved.
What's in it (only the verifier layer — the site comes from your branch)
sites/carmax/verify/verify_<0..19>.py— one deterministic verifier per tasksites/carmax/verify/verify_lib.py— shared helpers, same interface/format as the mergedsites/merriam_webster/verify/verify_lib.pysites/carmax/tasks.jsonl— addsverifier_path+judge_rubricto each taskEach verifier checks navigation (agent opened the right page — anti knowledge-shortcut),
DB after-state for stateful tasks (reservation / test-drive / saved / order / appraisal
vs the seed), and the answer against frozen ground truth. Rubrics carry the same concrete
values so the LLM judge can grade too.
Validated against the official react agent (
agent_demo/agent.py): verifier and LLM judgeagree except where the verifier is the more correct one (it caught an empty final answer and
a wrong-year reservation the LLM judge passed). Note: task aiming-lab#3 (Tesla Model 3 under 50k mi)
correctly fails as unsolvable, and a few site fixes are noted in the PR aiming-lab#24 review comments.