Add IGN mirror + task verifiers (site by @Edlison, verifiers by reviewer)#69
Open
Django-Jiang wants to merge 2 commits into
Open
Add IGN mirror + task verifiers (site by @Edlison, verifiers by reviewer)#69Django-Jiang wants to merge 2 commits into
Django-Jiang wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review: IGN mirror (PR #61) — reviewer verifiers + grading contract
Reviewed on branch
Edlison/main, builtwebharbor:ign-review, ran on alt ports(
:8204/41300-41316). IGN is site index 16 (port 40016).Mechanical checks: PASS
41300-41316)_health→ 109 items, 4 users, ok)md5 = da9379739f89373db60e2c32e801bc60(instance == instance_seed), stable across repeated reset + full task-drivingPOST /reset-all→ 17/17 ready in <1s.assets-revisionpinned torefs/pr/41;fetch_assets.sh ignsucceeds (paired HF PR: dataset discussions/41). Maintainer must bump to the merged SHA once the HF PR lands.Visual fidelity: PASS
/static/images/content/), IGN-style dark rail + cards.Functional depth: PASS (driven via real Playwright form submissions, confirmed against the DB)
TestPass123!)Task quality: 2 issues found — both FIXED in this PR
IGN--5 was infeasible (BLOCKER, fixed). The task requires favorite platform =
Nintendo Switch 2, butaccount_edit.html's<select name="favorite_platform">onlyoffered
PC / PlayStation 5 / Xbox Series X/S / Nintendo Switch / Mobile— no agent couldset "Nintendo Switch 2". Fix applied: added
Nintendo Switch 2to that select. The taskis now completable end-to-end (verified). (The
regionhalf is free text and already worked.)IGN--16 was pre-satisfied by the seed (WEAK, fixed).
seed_benchmark_usersalready putthe FF7 Rebirth video in alice's playlist as
queued— the task's exact target — so theafter-state couldn't confirm the "add". Fix applied: retargeted IGN--16 to the "Dragon
Ball Xenoverse 2 – Official Future Saga Chapter 4 Trailer" video, which is NOT in alice's
seed playlist, so the DB check is now discriminating (no-op FAILs, genuine add PASSes).
Minor (non-blocking, left as-is): IGN--0's story title exists as both an article (id 1) and a
same-title video (id 40); only the article is saveable (the video page has no save form), so
"open the story" means the article. Search matches all
video-type items on any querycontaining the token "video" — heavy distractors, but IGN--17 is NOT leaked (Platforms render
only on the detail page; the "Switch 2" text on the search page is an unrelated distractor
video title).
Grading contract authored (reviewer)
sites/ign/verify/verify_lib.py(shared deterministic + anchored-LLM utilities) + 19per-task verifiers
verify_0.py … verify_18.py.tasks.jsonlbackfilled withverifier_path+judge_rubricper row; noanswerkey.SQLite after-state (saved folder/note, playlist status, comment body, guide-checkpoint
completed, alert active flag, profile fields). LLM/screenshot checks are anchored on ground
truth and SKIP (never fail-close) when the LLM is unavailable.
Grading validation
after the fixes above.
eval_judge.py --verifier Trueruns the task's verifier fromverifier_pathand writeseval.json(confirmed pass=True on a completed run).md5unchanged).Verdict
Grading contract is ready and both task-quality issues are fixed in-PR. All 19 tasks are
feasible and deterministically gradable.