Answer "which model is best" on the homepage and check outbound links in CI

.agents/skills/write-homepage-answer/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: "write-homepage-answer"
+description: "Rewrite the hand-written answer block on the SenseBench homepage from current leaderboard data. Use when tools/check_homepage_answer.py reports it stale, or after runs that change the top of the board."
+---
+# Write the Homepage Answer Block
+
+**Version**: 1
+
+## Goal
+
+Rewrite the `<section class="answer-block">` in `src/sensebench/site/templates/index.html.j2` so
+that it answers "which LLM is best at word sense disambiguation" truthfully from current data.
+
+This block is deliberately hand-written rather than template-generated. Its interesting sentences —
+which models are statistically tied, what separates ConSeC from LENS — are judgements a format
+string cannot make. That is also why it can go stale, and why every figure in it is machine-checked.
+
+## Inputs
+
+* `$ARGUMENTS` — optional. A results directory; defaults to `results/`.
+
+## Context
+
+Read before starting:
+
+* `src/sensebench/site/templates/index.html.j2` — the current block, and the `.intro` / `.note`
+  classes it reuses
+* `src/sensebench/leaderboard/baselines.py` — baseline labels and, critically, the `source_note`
+  provenance caveats
+* `src/sensebench/leaderboard/schemes.py` — the nine schemes and `DEFAULT_SCHEME_ID`
+* `tools/check_homepage_answer.py` — the checks the result must pass
+
+## Steps
+
+1. Rebuild the data. Do not read figures from `_site/` without rebuilding it first; a checked-out
+   `_site/` can be months stale and may predate whole baselines.
+
+   ```bash
+   uv run sensebench site build --results-dir results --output-dir _site --strict
+   ```
+
+2. Read `_site/data/leaderboard.json`. Take the top entry's accuracy, both confidence bounds,
+   `correct_count`, `item_count`, `display_label`, `reasoning_effort` and `prompt_id`; the baseline
+   accuracies; and `summary.verified_run_count` / `summary.model_count`.
+
+3. **Compute the statistical ties.** Load `results/<run-id>/predictions.jsonl` for the top run and
+   for every run within a few points of it, and run an exact two-sided McNemar test over the
+   discordant pairs. Report which models are indistinguishable from the leader at p >= 0.05, and
+   name the notable models that are *not*. Accuracy proximity is not a tie — runs 0.4 points apart
+   have tested both ways.
+
+4. Take the latest run date as `max(created_at)` across entries, for both the prose and the
+   `<time datetime="...">` attribute.
+
+5. Write the block: an `<h2>` phrased as the question, one `<p class="intro">` paragraph, and a
+   `<p class="note">` counts line. Keep the five links and their targets.
+
+6. Verify:
+
+   ```bash
+   uv run sensebench site build --results-dir results --output-dir _site --strict
+   uv run python tools/check_homepage_answer.py
+   ```
+
+## Rules
+
+* **Recompute every figure.** Never carry a number over from the previous version of the block.
+* **Always name the scheme.** Accuracy here is meaningless without gold source and granularity. The
+  same run swings 95.60% to 85.39% between `lexen_fine` and `raganato_fine`, and the top model
+  changes with it. The block states default-scheme figures, so it must say so.
+* **Disclose ownership, and give the mechanism.** Glite LENS outscores the third-party supervised
+  baselines. Say so — but say *why*: it is retrained on model-relabelled SemCor, where ConSeC is
+  trained on the original human labels. Note that `baselines.py` records LENS's lexEN score as
+  "confirmatory under the paper's Section 6.4 rule", because its training labels share a model
+  family with the lexEN triage. It is not a clean independent comparison and must not be framed as
+  one.
+* **Prefer the number that goes down.** The residual error, the hard-subset score and the
+  supervised-baseline gap are more informative, and more defensible, than a headline near ceiling.
+* Keep it one paragraph, roughly 110–130 words: dated, numeric, source-attributed, self-contained.
+  It should stand alone if lifted verbatim, because it will be.
+* Match the surrounding prose: British spelling, no marketing register, no superlatives that are not
+  measurements.
+
+## Forbidden
+
+* NEVER state a figure that `tools/check_homepage_answer.py` cannot verify against built site data.
+* NEVER repeat a claim from the paper without re-testing it. The paper's "the top three families are
+  statistically indistinguishable" was true when written and is false on current data — as of
+  2026-07-25 only Claude Fable 5 ties with GPT-5.5 (p = 0.20), while Gemini 3.1 Pro (p = 0.031),
+  GPT-5.6 (p = 0.019) and Claude Opus 5 (p = 0.004) all differ significantly.
+* NEVER call any Glite artifact "best" or "strongest" without the training-data qualifier.
+* NEVER add a sixth link, and never link the same target twice.
+* NEVER rewrite the rest of the page "for AI readability". The one end-to-end study on this measured
+  body-only optimisation as worse: -6% citations and -16% top-10 presence after reranking. Add the
+  block; leave the intro, the tiles and the table alone.
+* NEVER edit `_site/` — it is build output.

.github/workflows/ci.yml

@@ -19,6 +19,24 @@ jobs:
       - run: uv run pytest -q
       - run: uv run python tools/verify_prompt.py --all
 
+  links:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      # Scoped to the files that carry outbound links we control. The Glite LENS 404 lived
+      # in a Python string, not a markdown link, so baselines.py is included deliberately.
+      - uses: lycheeverse/lychee-action@v2
+        with:
+          # --exclude-all-private drops the localhost vLLM example; the second exclude drops
+          # the literal `http://...` placeholder in the self-hosted docs.
+          args: >-
+            --no-progress --max-retries 3 --exclude-all-private
+            --exclude '^http://\.\.\.'
+            README.md docs src/sensebench/leaderboard/baselines.py
+          fail: true
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
   package:
     runs-on: ubuntu-latest
     steps:

.github/workflows/pages.yml

@@ -24,6 +24,7 @@ jobs:
           python-version: "3.12"
       - run: uv sync
       - run: uv run sensebench site build --results-dir results --output-dir _site --strict
+      - run: uv run python tools/check_homepage_answer.py
       - name: Upload pull request preview artifact
         if: github.event_name == 'pull_request'
         uses: actions/upload-artifact@v4

src/sensebench/site/build.py

@@ -434,6 +434,18 @@ def _run_date_label(created_at: str) -> str:
     return datetime.fromisoformat(created_at).date().isoformat()
 
 
+def _index_description(*, summary: SiteSummary) -> str:
+    dataset_labels = ", ".join(
+        _dataset_version_label(version) for version in summary.dataset_versions
+    )
+    dataset_clause = "" if len(dataset_labels) == 0 else f" on {dataset_labels}"
+    return (
+        "Verified leaderboard for English word sense disambiguation: "
+        f"{summary.verified_run_count} audited runs across {summary.model_count} models"
+        f"{dataset_clause}. Top accuracy {_format_percent(summary.top_accuracy)}."
+    )
+
+
 def _run_page_title(entry: LeaderboardEntry) -> str:
     headline = (
         f"{entry.display_label or entry.model} {WSD_TASK_PHRASE}"
@@ -1644,8 +1656,8 @@ def _render_index(
         env=env,
         template_name="index.html.j2",
         base_url=base_url,
-        title="SenseBench Leaderboard",
-        description="Verified leaderboard for English word sense disambiguation with LLMs.",
+        title="SenseBench (WSD): LLM Word Sense Disambiguation Leaderboard",
+        description=_index_description(summary=site_data.summary),
         path=path,
         context={
             SITE_DATA_CONTEXT_KEY: site_data,

src/sensebench/site/templates/index.html.j2

@@ -33,6 +33,33 @@
   </dl>
 </section>
 
+{#
+  Hand-written by an agent, not generated: the statistical-tie set and the ConSeC/LENS
+  training-data distinction are judgements a format string cannot make. Every figure is
+  guarded by test_homepage_answer_block_figures_match_the_leaderboard in tests/test_site.py,
+  so stale numbers fail CI. Rewrite with .agents/skills/write-homepage-answer.
+#}
+<section class="answer-block" aria-labelledby="answer-heading">
+  <h2 id="answer-heading">Which LLM is best at word sense disambiguation?</h2>
+  <p class="intro">
+    As of <time datetime="2026-07-25">25 July 2026</time>, the best verified result on
+    <a href="https://github.com/GliteTech/lexen" rel="noopener">lexEN v1</a> is
+    <strong>95.60%</strong> (95% CI 95.00–96.17), from
+    <a href="{{ base_path }}runs/gpt-5.5-xhigh-reasoning-p001-lexen-v1-20260617/">GPT-5.5 at xhigh
+    reasoning effort</a> under <a href="{{ base_path }}prompts/p001/">registered prompt p001</a> —
+    4,647 of 4,861 polysemous English items. Only Claude Fable 5 is statistically
+    indistinguishable from it (95.21%, McNemar p = 0.20); Gemini 3.1 Pro, GPT-5.6 and Claude
+    Opus 5 all fall significantly below. WordNet's most-frequent-sense heuristic scores 61.55%
+    on the same items; among supervised systems ConSeC, trained on the original human labels,
+    reaches 84.88%, while Glite's own LENS, retrained on model-relabelled SemCor, reaches
+    89.69%. Figures use the default labels — lexEN v1 gold at
+    <a href="{{ base_path }}label-schemes/">WordNet fine granularity</a>;
+    <a href="{{ base_path }}coarsening/">coarser sense inventories</a> score substantially
+    higher. Every number is recomputed in CI from the stored raw API responses.
+  </p>
+  <p class="note">198 verified runs · 63 models · latest run 25 July 2026</p>
+</section>
+
 <section class="scheme-bar" aria-label="Score labels">
   <div class="scheme-bar-controls">
     <label class="scheme-select">

tests/test_site.py

@@ -20,6 +20,7 @@
     GPU_REFERENCE_HOURLY_RATE_USD,
     GPU_REFERENCE_RATES_AS_OF,
 )
+from sensebench.leaderboard.schemes import DEFAULT_SCHEME_ID
 from sensebench.paths import (
     CALLS_FILENAME,
     CNAME_FILENAME,
@@ -167,6 +168,11 @@
 ALTERNATE_QUANTIZATION: str = "awq-int4"
 RERUN_CREATED_AT: str = "2026-07-20T00:00:00+00:00"
 RERUN_DATE_LABEL: str = "2026-07-20"
+ANSWER_BLOCK_MARKER: str = '<section class="answer-block"'
+ANSWER_HEADING_TEXT: str = "Which LLM is best at word sense disambiguation?"
+TITLE_HEAD_TERM: str = "Word Sense Disambiguation Leaderboard"
+DESCRIPTION_RUN_COUNT_TEXT: str = "1 audited runs"
+LEXEN_FINE_SCHEME_ID: str = "lexen_fine"
 LARGE_SHARE_CARD_TEXT: str = '<meta name="twitter:card" content="summary_large_image">'
 SMALL_SHARE_CARD_TEXT: str = '<meta name="twitter:card" content="summary">'
 OG_IMAGE_ABSOLUTE_URL: str = f"{TEST_BASE_URL}{SITE_ASSETS_DIRNAME}/{OG_IMAGE_FILENAME}"
@@ -736,6 +742,38 @@ def _page_title(html_text: str) -> str:
     return html_text[start : html_text.index(TITLE_CLOSE_TAG, start)]
 
 
+def test_homepage_answers_the_question_in_static_html(
+    tmp_path: Path,
+    monkeypatch: MonkeyPatch,
+) -> None:
+    dataset = _dataset()
+    _patch_registered_dataset(monkeypatch=monkeypatch, dataset=dataset)
+    results_dir = tmp_path / SUBMITTED_RESULTS_DIR
+    output_dir = tmp_path / SITE_OUTPUT_DIR
+    _write_verified_run(results_dir=results_dir, dataset=dataset, run_id=TEST_RUN_ID)
+
+    build_site(
+        results_dir=results_dir,
+        output_dir=output_dir,
+        base_url=TEST_BASE_URL,
+        strict=True,
+    )
+
+    index_html = (output_dir / INDEX_HTML_FILENAME).read_text(encoding="utf-8")
+
+    assert ANSWER_BLOCK_MARKER in index_html
+    assert ANSWER_HEADING_TEXT in index_html
+    assert TITLE_HEAD_TERM in _page_title(index_html)
+    # The description is generated, so it must carry live counts rather than prose.
+    assert DESCRIPTION_RUN_COUNT_TEXT in index_html
+
+
+def test_answer_block_scheme_qualifier_still_names_the_default() -> None:
+    # The block states default-scheme figures in prose. If the default moves, the prose
+    # silently becomes false, so pin it here rather than discovering it on the homepage.
+    assert DEFAULT_SCHEME_ID == LEXEN_FINE_SCHEME_ID
+
+
 def test_pages_advertise_a_large_share_card(
     tmp_path: Path,
     monkeypatch: MonkeyPatch,

tools/check_homepage_answer.py

@@ -0,0 +1,125 @@
+"""Verify the homepage answer block still matches the leaderboard.
+
+The block in `index.html.j2` is written by an agent rather than generated, so its figures
+can drift as runs are merged. This checks every claim in it against the built site data and
+fails when one is stale. Rewrite the block with `.agents/skills/write-homepage-answer`.
+
+Run against a built site:
+
+    uv run sensebench site build --results-dir results --output-dir _site --strict
+    uv run python tools/check_homepage_answer.py
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from argparse import ArgumentParser
+from dataclasses import dataclass
+from pathlib import Path
+
+INDEX_HTML: Path = Path("_site/index.html")
+LEADERBOARD_JSON: Path = Path("_site/data/leaderboard.json")
+
+BLOCK_PATTERN: re.Pattern[str] = re.compile(
+    r'<section class="answer-block".*?</section>', re.DOTALL
+)
+PERCENT_PATTERN: re.Pattern[str] = re.compile(r"(\d+\.\d{2})%")
+COUNT_PATTERN: re.Pattern[str] = re.compile(r"(\d{1,3}(?:,\d{3})+|\b\d{2,4}\b)")
+TIME_PATTERN: re.Pattern[str] = re.compile(r'<time datetime="(\d{4}-\d{2}-\d{2})"')
+
+REQUIRED_BASELINE_LABELS: tuple[str, ...] = ("MFS (WordNet first sense)", "ConSeC", "Glite LENS")
+
+
+@dataclass(frozen=True, slots=True)
+class Failure:
+    claim: str
+    detail: str
+
+
+def _pct(value: float) -> str:
+    return f"{value * 100:.2f}"
+
+
+def _check(*, index_html: Path, leaderboard_json: Path) -> list[Failure]:
+    html_text = index_html.read_text(encoding="utf-8")
+    match = BLOCK_PATTERN.search(html_text)
+    if match is None:
+        return [Failure("answer block", f'no <section class="answer-block"> in {index_html}')]
+    block = match.group(0)
+
+    payload = json.loads(leaderboard_json.read_text(encoding="utf-8"))
+    entries = payload["entries"]
+    baselines = {row["label"]: row for row in payload.get("baselines", [])}
+    summary = payload["summary"]
+    top = entries[0]
+
+    failures: list[Failure] = []
+
+    # Every figure the prose is allowed to state, derived from live data.
+    allowed = {_pct(entry["accuracy"]) for entry in entries if entry.get("accuracy") is not None}
+    allowed |= {_pct(row["accuracy"]) for row in baselines.values() if row.get("accuracy")}
+    allowed |= {_pct(top["accuracy_ci"]["low"]), _pct(top["accuracy_ci"]["high"])}
+
+    for stated in PERCENT_PATTERN.findall(block):
+        if stated not in allowed:
+            failures.append(
+                Failure(f"{stated}%", "matches no current run, baseline or confidence bound")
+            )
+
+    def require(label: str, value: str) -> None:
+        if value not in block:
+            failures.append(Failure(label, f"expected {value!r}, not present in the block"))
+
+    require("top accuracy", f"{_pct(top['accuracy'])}%")
+    require("CI low", f"{_pct(top['accuracy_ci']['low'])}")
+    require("CI high", f"{_pct(top['accuracy_ci']['high'])}")
+    require("top model", top["display_label"] or top["model"])
+    require("correct count", f"{top['correct_count']:,}")
+    require("item count", f"{top['item_count']:,}")
+    require("run count", str(summary["verified_run_count"]))
+    require("model count", str(summary["model_count"]))
+
+    for label in REQUIRED_BASELINE_LABELS:
+        row = baselines.get(label)
+        if row is None:
+            failures.append(Failure(f"baseline {label}", "absent from the built site data"))
+            continue
+        require(f"baseline {label}", f"{_pct(row['accuracy'])}%")
+
+    latest_run_date = max(entry["created_at"] for entry in entries)[:10]
+    stated_dates = TIME_PATTERN.findall(block)
+    if len(stated_dates) == 0:
+        failures.append(Failure("<time>", "the block states no machine-readable date"))
+    elif stated_dates[0] != latest_run_date:
+        failures.append(
+            Failure("<time>", f"states {stated_dates[0]}, latest run is {latest_run_date}")
+        )
+
+    return failures
+
+
+def main() -> int:
+    parser = ArgumentParser(description=__doc__)
+    parser.add_argument("--index-html", type=Path, default=INDEX_HTML)
+    parser.add_argument("--leaderboard-json", type=Path, default=LEADERBOARD_JSON)
+    args = parser.parse_args()
+
+    for path in (args.index_html, args.leaderboard_json):
+        if not path.exists():
+            parser.error(f"{path}: not found; run `sensebench site build` first")
+
+    failures = _check(index_html=args.index_html, leaderboard_json=args.leaderboard_json)
+    if len(failures) > 0:
+        print(f"{args.index_html}: homepage answer block is stale")
+        for failure in failures:
+            print(f"  {failure.claim}: {failure.detail}")
+        print("\nRewrite it with .agents/skills/write-homepage-answer, then rebuild the site.")
+        return 1
+
+    print(f"{args.index_html}: homepage answer block matches the leaderboard")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())