Skip to content

Add IGN mirror + task verifiers (site by @Edlison, verifiers by reviewer)#69

Open
Django-Jiang wants to merge 2 commits into
aiming-lab:mainfrom
Django-Jiang:review/ign-verifiers
Open

Add IGN mirror + task verifiers (site by @Edlison, verifiers by reviewer)#69
Django-Jiang wants to merge 2 commits into
aiming-lab:mainfrom
Django-Jiang:review/ign-verifiers

Conversation

@Django-Jiang

Copy link
Copy Markdown
Contributor

Review: IGN mirror (PR #61) — reviewer verifiers + grading contract

Reviewed on branch Edlison/main, built webharbor:ign-review, ran on alt ports
(:8204 / 41300-41316). IGN is site index 16 (port 40016).

Mechanical checks: PASS

  • All 17 sites return 200 (41300-41316)
  • Control plane healthy (_health → 109 items, 4 users, ok)
  • Byte-identical reset: md5 = da9379739f89373db60e2c32e801bc60 (instance == instance_seed), stable across repeated reset + full task-driving
  • POST /reset-all → 17/17 ready in <1s
  • Assets: .assets-revision pinned to refs/pr/41; fetch_assets.sh ign succeeds (paired HF PR: dataset discussions/41). Maintainer must bump to the merged SHA once the HF PR lands.

Visual fidelity: PASS

  • Home renders 36/36 real content images (all under /static/images/content/), IGN-style dark rail + cards.
  • Detail / listing / reviews / videos / guides pages populate correctly.

Functional depth: PASS (driven via real Playwright form submissions, confirmed against the DB)

  • Login (all 4 benchmark users, TestPass123!)
  • Save story with folder + note; playlist add with status + note; comment; guide-checklist completion; alert create + pause; remove saved; profile region edit — all persist to SQLite.

Task quality: 2 issues found — both FIXED in this PR

  1. IGN--5 was infeasible (BLOCKER, fixed). The task requires favorite platform =
    Nintendo Switch 2, but account_edit.html's <select name="favorite_platform"> only
    offered PC / PlayStation 5 / Xbox Series X/S / Nintendo Switch / Mobile — no agent could
    set "Nintendo Switch 2". Fix applied: added Nintendo Switch 2 to that select. The task
    is now completable end-to-end (verified). (The region half is free text and already worked.)

  2. IGN--16 was pre-satisfied by the seed (WEAK, fixed). seed_benchmark_users already put
    the FF7 Rebirth video in alice's playlist as queued — the task's exact target — so the
    after-state couldn't confirm the "add". Fix applied: retargeted IGN--16 to the "Dragon
    Ball Xenoverse 2 – Official Future Saga Chapter 4 Trailer" video, which is NOT in alice's
    seed playlist, so the DB check is now discriminating (no-op FAILs, genuine add PASSes).

Minor (non-blocking, left as-is): IGN--0's story title exists as both an article (id 1) and a
same-title video (id 40); only the article is saveable (the video page has no save form), so
"open the story" means the article. Search matches all video-type items on any query
containing the token "video" — heavy distractors, but IGN--17 is NOT leaked (Platforms render
only on the detail page; the "Switch 2" text on the search page is an unrelated distractor
video title).

Grading contract authored (reviewer)

  • sites/ign/verify/verify_lib.py (shared deterministic + anchored-LLM utilities) + 19
    per-task verifiers verify_0.py … verify_18.py.
  • tasks.jsonl backfilled with verifier_path + judge_rubric per row; no answer key.
  • Deterministic-first: navigation evidence (anti knowledge-shortcut) + on-page answer match +
    SQLite after-state (saved folder/note, playlist status, comment body, guide-checkpoint
    completed, alert active flag, profile fields). LLM/screenshot checks are anchored on ground
    truth and SKIP (never fail-close) when the LLM is unavailable.

Grading validation

  • No-op (homepage only, empty answer, DB = seed): all 19 verifiers FAIL. Zero false positives.
  • Genuine completion (real form-driven actions): all 19 PASS — including IGN--5 and IGN--16
    after the fixes above.
  • Wrong answer (read-only 4/17 naming the wrong item): FAIL.
  • Shortcut (correct answer, no detail-page nav): FAIL on the nav gate.
  • State-mismatch (correct nav, DB reverted to seed): FAIL on the DB gate.
  • eval_judge.py --verifier True runs the task's verifier from verifier_path and writes
    eval.json (confirmed pass=True on a completed run).
  • Byte-identical reset holds after the site fix + all task-driving (md5 unchanged).

Verdict

Grading contract is ready and both task-quality issues are fixed in-PR. All 19 tasks are
feasible and deterministically gradable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants