Skip to content

Add Kaggle mirror + task verifiers (site by @KaKituken, verifiers by reviewer)#70

Open
Django-Jiang wants to merge 1 commit into
aiming-lab:mainfrom
Django-Jiang:review/kaggle-verifiers
Open

Add Kaggle mirror + task verifiers (site by @KaKituken, verifiers by reviewer)#70
Django-Jiang wants to merge 1 commit into
aiming-lab:mainfrom
Django-Jiang:review/kaggle-verifiers

Conversation

@Django-Jiang

Copy link
Copy Markdown
Contributor

Review: Kaggle mirror (PR #54) — reviewer verifiers + grading contract

Reviewed PR #54, rebased the site onto current main, built the 17-site image
webharbor:kaggle-rebase, ran on alt ports. Kaggle is registered at index 16 (port 40016).

Mechanical checks: PASS

  • All 16 sites return 200 (41400-41415)
  • Control plane healthy (_health → 17 competitions, 16 datasets, ok)
  • Byte-identical reset: md5 = 131cdab6ed840e4e74324ab5bd24dda4 (instance == instance_seed), stable after full task-driving + resets
  • POST /reset-all → all sites ready
  • Assets pinned to refs/pr/37 (HF discussions/37); fetch_assets.sh succeeds. Maintainer must bump to the merged SHA once the HF PR lands.

Visual / functional depth: PASS (driven via real Playwright form/JS actions, confirmed against the DB)

Login (all 4 benchmark users, TestPass123!); join competition (+ team name); upvote; bookmark;
follow; new discussion; post comment; profile-location edit; dataset download — all persist to SQLite.

Task quality: 1 verifier-side fix, otherwise sound

  1. Kaggle--11 was pre-satisfied by the seed (fixed in the verifier). seed_discussions already
    seeds one davidtran comment on the titanic-feature-ideas thread, so an existence check
    couldn't confirm the agent's new comment. The verifier now uses a delta (davidtran comment
    count on that thread must increase vs the seed). No task-text change needed.

Minor (non-blocking): Kaggle--0 ("Titanic passenger survival") — two Getting-Started Titanic
competitions exist (titanic-survival, spaceship-titanic-rescue), but both use metric
"Classification Accuracy", so the answer is unambiguous; the verifier requires navigating the
titanic-survival page. Several read-only answers (accuracy, ROC AUC, Apache 2.0, CC0, 7 lessons)
are frontier-LLM-guessable; the anti-shortcut nav gate (must open the specific competition/model/
course/dataset page) mitigates this, and site-specific tasks (team names, host schedules,
carolwong's notebook, rankings) anchor the harder ones.

Rebased onto current main in this PR

PR #54 branched from a 16-site main and registered kaggle at index 15 (a slot now taken by
merriam_webster). This reviewer PR rebases the site onto current main:

  • kaggle re-registered as index 16 / port 40016 in websyn_start.sh, control_server.py, and
    Dockerfile (EXPOSE 40000-40016).
  • .assets-revision pinned to refs/pr/37 (HF discussions/37), which contains all 16 current-main
    site tarballs plus kaggle.tar.gz — so this PR builds end-to-end. Maintainer: after HF discussions/37
    is rebased onto HF main and merged, bump .assets-revision to the merged SHA.
  • The full 17-site image builds; all 17 sites return 200; kaggle serves at 40016; byte-identical reset
    holds; the verifiers were re-confirmed against the rebased container (read-only PASS, no-op FAIL,
    and a stateful drive+PASS at the new port). The current-main agent_demo provides the
    eval_judge --verifier True PRIMARY-grader path.

Grading contract authored (reviewer)

  • sites/kaggle/verify/verify_lib.py (shared deterministic + anchored-LLM utilities) + 20 per-task
    verifiers verify_0.py … verify_19.py.
  • tasks.jsonl backfilled with verifier_path + judge_rubric per row; no answer key.
  • Deterministic-first: navigation evidence (anti knowledge-shortcut) + on-page answer match +
    SQLite after-state (competition_entries, votes, bookmarks, follows, discussions/comments, download
    counter, user profile). LLM/screenshot checks anchored on ground truth and SKIP when the LLM is
    unavailable.

Grading validation (verifiers run standalone against the live container DB)

  • No-op (homepage only, empty answer, DB = seed): all 20 verifiers FAIL. Zero false positives.
  • Genuine completion (real form/JS-driven actions + correct read-only answers): all 20 PASS.
  • Wrong answer (read-only naming the wrong metric/license/user/team): FAIL.
  • Shortcut (correct answer, no detail-page nav): FAIL on the nav gate.
  • State-mismatch (correct nav, DB reverted to seed): FAIL on the DB gate.
  • Byte-identical reset holds after the full drive (md5 unchanged).

Verdict

Grading contract is ready and validated; the site is rebased onto current main (kaggle at index
16 / port 40016) and builds from refs/pr/37. Ready to merge once the maintainer bumps
.assets-revision to the merged HF SHA after HF discussions/37 lands on HF main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant