Add NVIDIA mirror + task verifiers (site by @KaKituken, verifiers by reviewer)#58
Open
DEM1TASSE wants to merge 2 commits into
Open
Add NVIDIA mirror + task verifiers (site by @KaKituken, verifiers by reviewer)#58DEM1TASSE wants to merge 2 commits into
DEM1TASSE wants to merge 2 commits into
Conversation
New WebHarbor mirror of nvidia.com on port 40015 (16th site). - sites/nvidia/: Flask + SQLAlchemy app — GPU/hardware catalog (Product with full spec sheet, Article, Driver, Review, CartItem, Order, WishlistItem, NewsletterSubscriber). Routes: browse/filter, product detail with specs, spec-comparison tool, driver-download finder, news, scored token-overlap search (all listings + global), auth, account, cart/checkout, wishlist, reviews, newsletter. 22 Jinja2 templates, NVIDIA black + green design. - 25 real products across 5 categories (GeForce RTX 50/40, RTX PRO / Ada workstation, data-center H200/H100/A100/B200/GH200/L40S, Jetson, SHIELD) with real published specs; 8 real NVIDIA Newsroom articles; 9 drivers. - Idempotent seeding (function-level gates) with the 4 canonical benchmark users (alice.j@test.com et al.); byte-identical reset verified. - 20 WebVoyager tasks in sites/nvidia/tasks.jsonl. - Registered in websyn_start.sh, control_server.py, Dockerfile (EXPOSE ...40015). - Re-applies the .gitignore fix: scraped_data/ and instance/ patterns had inline comments that gitignore treats as part of the pattern. Heavy assets (instance_seed/nvidia.db, static/images/) ship via the ChilleD/WebHarbor HF dataset, not this repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer deliverable for the NVIDIA mirror (site by @KaKituken, PR aiming-lab#55): one deterministic verifier per task under sites/nvidia/verify/, plus verifier_path + judge_rubric recorded in every tasks.jsonl row. No answer key in tasks.jsonl — ground truth lives only inside the verifiers. Design is deterministic-first: (1) trajectory navigation (anti knowledge- shortcut), (2) SQLite DB after-state for the 8 stateful tasks (cart / order / wishlist / review / profile / newsletter), (3) answer vs frozen ground truth, with the LLM only as an anchored consistency check. Validated against the official react agent (agent_demo/agent.py): a no-op run fails all 20 verifiers, a correct-answer-without-navigation run fails on the nav gate, and genuine completions pass. On the full 20-task run the verifier and LLM judge agreed 18/20; both divergences were the LLM judge false- negativing a genuine success the verifier caught via DB after-state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the NVIDIA mirror + its task verifiers into
mainas one PR, following the samefork-of-fork pattern used for CarMax (#57). Companion to the site PR #55.
Authorship is preserved — both contributors stay credited. This branch is @KaKituken's
NVIDIA site commit with the reviewer's verifier commit on top; each commit keeps its original
author (site = @KaKituken, verifiers = the reviewer). Please merge with a regular merge (not
squash) so both signatures are kept.
Contents
sites/nvidia/site (Flask app, templates, seed, 20 tasks) — @KaKituken (from Add NVIDIA mirror site #55)sites/nvidia/verify/verify_<0..19>.py+verify_lib.py, andverifier_path+judge_rubricin every
tasks.jsonlrow — reviewer deliverable. Noanswerkey in tasks.jsonl; groundtruth lives only inside the verifiers.
Verifier design & validation
Deterministic-first per task: navigation gate (anti knowledge-shortcut) + SQLite DB after-state
for the 8 stateful tasks (cart / order / wishlist / review / profile / newsletter) + answer vs
frozen ground truth, with the LLM only as an anchored consistency check. Validated against the
official react agent (
agent_demo/agent.py): no-op fails all 20; correct-answer-without-navfails the nav gate; genuine completions pass. On the full 20-task run the verifier and LLM judge
agreed 18/20, and both divergences were the LLM judge false-negativing a genuine success the
verifier caught via DB after-state.
Known conflict (please resolve on merge)
mainalready merged Merriam-Webster at index 15 / port 40015, and this branch (predating thatmerge) registers NVIDIA at the same slot. On merge, move NVIDIA to the next free slot — index
16 / port 40016 — in
websyn_start.sh,control_server.py,Dockerfile(EXPOSE …-40016),and bump the
webport insites/nvidia/tasks.jsonl. The verifiers are port-agnostic (theymatch URL paths, not the host port).
Companion to #55. A full review (mechanical / visual / functional / task-quality / grading
contract) is posted as a comment on #55.