Add Target mirror + task verifiers (site rebuilt from real target.com data)#72
Open
1171-jpg wants to merge 1 commit into
Open
Add Target mirror + task verifiers (site rebuilt from real target.com data)#721171-jpg wants to merge 1 commit into
1171-jpg wants to merge 1 commit into
Conversation
…erifiers Adds the Target mirror as the 17th site (index 16, port 40016), rebuilt from real target.com data scraped with Playwright. Catalog: 1,932 products across 18 categories, 562 brands, with real product photography, 19,821 specification rows and 2,350 scraped guest reviews. Tasks: 20 WebVoyager tasks in sites/target/tasks.jsonl, each anchored on a fact that only appears on a detail page. Grading contract: sites/target/verify/ with verify_lib.py and one deterministic verifier per task, recorded as verifier_path + judge_rubric on every task row. Ground truth lives only in the verifiers, never in tasks.jsonl. Assets uploaded to HF as target.tar.gz (discussions/46). Based on the Target site skeleton from aiming-lab#51 by @Lxr-max, which was reassigned for rebuild after review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PLRyFZ4TjY9pC7uZkKJC6H
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Target mirror as the 17th WebHarbor site, together with its 20 benchmark tasks
and the deterministic grading contract for them.
Supersedes #51, which was reassigned to me for rebuild. The catalog, imagery, copy, tasks and
verifiers here are all new.
Why a rebuild
The existing Target mirror could not be brought to standard by patching: its catalog was
carried over from another site's product set, all imagery was generated placeholders, and
the seed DB was not byte-identical reproducible. Each of those requires replacing the
content domain rather than amending it.
Mirrored site
Target — https://www.target.com/
17th WebHarbor site, served on port 40016.
Port assignment
maingives port 40015 tomerriam_webster, so Target takes index 16 / port 40016,registered in all four places:
websyn_start.sh,control_server.py,Dockerfile(EXPOSE … 40016) and thewebfield of everytasks.jsonlrow. The verifiersare port-agnostic — they match URL paths, not the host port.
Verified after the move: 17/17 sites healthy and returning 200, byte-identical reset holds
for both
targetandmerriam_webster, andreset-allleaves 17/17 alive.Seeded rows per major model
Products by category: Home Decor 120 · Women's Clothing 116 · Shoes 116 · Kids' Clothing 115 ·
Men's Clothing 115 · Bedding 112 · Toys 113 · Furniture 111 · Pets 109 · Beauty 108 ·
Personal Care 104 · Sports & Outdoors 102 · Baby 101 · Household Essentials 100 · Grocery 99 ·
Kitchen & Dining 98 · Video Games 98 · Electronics 95.
All product names, prices, ratings, rating counts, photographs, specification tables and guest
reviews are scraped from target.com. Nothing in the catalog is invented.
Site capabilities
ranked results.
/searchcarries a facet rail (category / brand / price / rating / deals /fulfilment / sort); each facet's options are computed from every other facet, so the rail
can never offer a combination that returns nothing.
Click-to-open rather than hover, because a browser-driving agent clicks reliably but hovers
fragilely;
/categoriescarries the same tree for agents that never open the menu.recommend", protection plans, store inventory and scraped guest reviews.
pricing, and says so explicitly when the selected items come from different categories.
pickup flows, completed checkout steps are navigable so an address can be corrected without
restarting, and the confirmation page carries the pickup store / pickup time / shipping
address / total.
requests, and product review submission. Forms validate server-side.
Tasks
20 tasks in
sites/target/tasks.jsonl(WebVoyager schema) across 10 functional areas:help centre, search + filter, product detail, multi-step comparison, account, orders, cart,
checkout (delivery and pickup), wish list, customer service, and user-generated reviews.
7 are stateful and verified against the database rather than the answer text. 4 require ≥7
steps (full delivery checkout, full pickup checkout, and two multi-page comparisons — e.g.
"across the Red Baron frozen pizzas, which has the lowest sodium per serving", which cannot
be answered without opening several product pages).
Answers are anchored on facts that appear only on a detail page — nutrition rows, specification
tables, per-attribute guest ratings, protection-plan pricing — and each was checked against the
rendered search-results DOM to confirm it does not leak there. Two tasks were rewritten during
this check: one asked for price and rating (both printed on the search card) and one asked for
a figure that the search page prints verbatim in the article summary.
There are no "ask me which one" tasks.
design-tasksencourages disambiguation tasks, butCONTRIBUTING rejects tasks that presuppose a human in the loop, and the
merriam_websterreference carries none — so the two disambiguation tasks drafted earlier were re-anchored onto
questions with a single correct answer.
Grading contract
sites/target/verify/—verify_lib.pyplus one verifier per task, withverifier_pathandjudge_rubricon everytasks.jsonlrow. Noanswerkey; ground truth lives only insidethe verifiers, and the rubrics were scanned against every answer value to confirm they state
rules rather than answers.
Deterministic-first: a navigation gate (a correct answer with no matching page visit is a
knowledge shortcut), answer-value extraction from free text, and SQLite after-state diffing for
the stateful tasks. LLM helpers are anchored on frozen ground truth and short-circuit under
--no_llm.Validation
41 scenarios across all 20 tasks; every one behaves as expected.
Target--17was additionallyvalidated against a real Playwright-driven run (real trajectory, real screenshots, real
before/after databases) rather than a synthetic fixture.
Byte-identical reset
seed_data.pyuses a fixedSEED_TIMESTAMPwith explicit timestamps on every row and no RNG,so consecutive builds produce an identical md5 — the models'
created_atcolumns default todatetime.utcnow, which would otherwise make the seed unreproducible. All 13seed_*()functions early-return as a whole. All 17 sites return 200, and
reset-allleaves 17/17 alive.Visual fidelity vs. the real site
Mirror rendered at
localhost:40016(compare against https://www.target.com/).More (categories, deals, compare, cart, pickup checkout)
Paired HuggingFace assets PR
https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/46 — uploads
target.tar.gz(instance_seed/target.db + static/images: 1,932 productphotos and 18 category images, 15 MB).
Merge the HF dataset PR before this one so a clean-clone build can fetch the assets;
.assets-revisionis pinned to that merge SHA in this branch.Notes for human review
behind a "Show more" control whose button resists programmatic clicking. Those categories
have real names, prices, ratings, photographs and full specification tables, but no prose
description. No task depends on one.
(0 of 8 sampled under the gentlest scrape settings) rather than assumed from a failed scrape.
Review-based tasks avoid that category.
placeholders, so store pages render without a hero image rather than with a fabricated one.
category (Roast Type, Flavor, …). The scraped specification data would support them, but it
needs per-category configuration and unit normalisation; the generic facets cover the tasks.
the picker lists them once and the store is chosen separately, rather than repeating 75
near-identical options.