Add Recreation.gov task verifiers (site by @Chubi-alt, verifiers by reviewer)#66
Open
Django-Jiang wants to merge 4 commits into
Open
Add Recreation.gov task verifiers (site by @Chubi-alt, verifiers by reviewer)#66Django-Jiang wants to merge 4 commits into
Django-Jiang wants to merge 4 commits into
Conversation
…eviewer) Add the reviewer grading contract for the Recreation.gov mirror (site + 20 tasks by @Chubi-alt, PR aiming-lab#8): - sites/recreation_gov/verify/verify_lib.py: shared deterministic + anchored-LLM utilities (trajectory navigation, answer match, SQLite after-state helpers). - verify_0.py .. verify_19.py: one deterministic verifier per task. - tasks.jsonl: backfill verifier_path + judge_rubric for all 20 tasks. Grading is deterministic-first: navigation evidence (anti knowledge-shortcut), answer/token match against frozen ground truth, and DB after-state checks for the stateful tasks (save / cart / checkout / cancel / account edit / register / review). LLM utilities are anchored on ground truth and used only where exact matching is brittle. Validated: a no-op run FAILs all 20; a correct solve PASSes all 20 (factual via synthetic trajectories, stateful via reset -> Playwright mutation -> verify); wrong-answer, no-navigation, and DB-unchanged cases FAIL as expected.
Address code-review findings on the recreation_gov grading contract: - verify_lib: LLM checks no longer fail-close. _verdict returns a None verdict when the model is unavailable and Judge.check skips it, so a correct answer is never failed just because no LLM was configured; a real LLM verdict still gates. count_present now matches tokens on word boundaries (e.g. "date" no longer matches inside "updated"). Removed unused helpers carried from the template (screenshot utilities, answer_equals, extract_ints, user_emails). - tasks.jsonl: rewrite judge_rubric entries to state grading rules only; drop the ground-truth answer values that had leaked into the agent-facing file. - Tighten deterministic gates: verify_18 requires naming both permit types (not the bare word "both"); verify_16 checks the required visit date; verify_13 no longer uses a brittle exact-list status compare. verify_5's permit-vs-campground classification is LLM-arbitrated (a blunt substring ban wrongly failed the correct "a permit, not a campground" phrasing). Validated against a live instance: a no-op run FAILs all 20; changed gates PASS a correct answer and FAIL the specific wrong case; stateful tasks pass via real mutation; container reset remains byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reviewer grading contract for the Recreation.gov mirror. Based on @Chubi-alt's original PR #8 (site + 20 tasks); this branch stacks on that base and adds the deterministic verifiers +
judge_rubricthat the reviewer role is responsible for. Ground truth lives only inside the verifiers —tasks.jsonlgetsverifier_path+judge_rubriconly (no answer key).Review summary
Mechanical: PASS
recreation_gov(md5 ofinstance/vsinstance_seed/match)reset-all~0.7sVisual + functional: PASS
Task quality: PASS — no task changes required
quesfor all auth tasks; no no-op stateful tasks (every target state differs from seed); no answer leaks on listing cardsGrading contract added
sites/recreation_gov/verify/verify_lib.py+verify_0.py…verify_19.py(one per task)tasks.jsonl:verifier_path+judge_rubricbackfilled for all 20 tasksGrading validated three ways
eval_judge.py --verifier Trueentry point.