From 850d754e65b83c0809ae351dd2905c8977fb2ee9 Mon Sep 17 00:00:00 2001 From: Vassili Philippov Date: Sun, 26 Jul 2026 13:20:14 +0100 Subject: [PATCH] Answer "which model is best" on the homepage and check outbound links in CI Nothing on the site answered the question people actually ask. The top accuracy existed only as an unattributed stat tile that a reader -- or a retrieval system -- had to join to a table row across 200 KB of HTML, and the homepage title was the generic "SenseBench Leaderboard". The homepage now opens the leaderboard with an H2 phrased as the question and one self-contained paragraph answering it: the top verified result with its confidence interval, the run and prompt that produced it, which models are statistically tied with it, and where the MFS, ConSeC and LENS baselines sit. Five links carry a reader from each claim to the artifact behind it. The paragraph is written by hand rather than generated. Its interesting sentences are judgements a format string cannot make: which models tie is an exact McNemar test over stored per-item predictions, not accuracy proximity -- runs 0.4 points apart have tested both ways -- and the gap between ConSeC and LENS is a training-data difference worth explaining rather than a ranking worth asserting. Only Claude Fable 5 currently ties with GPT-5.5 at p = 0.20; the paper's "top three families are statistically indistinguishable" no longer holds on current data. Hand-written prose goes stale, so tools/check_homepage_answer.py verifies every figure in the block against the built site data and fails when one drifts, and .agents/skills/write-homepage-answer records how to rewrite it -- including the traps: recompute rather than copy, always name the label scheme, and never call a Glite artifact "strongest" without the qualifier. Accuracy here is meaningless without its scheme. The same run scores 95.60% under lexEN fine and 85.39% under Raganato fine, and the top model changes with it, so the prose names its scheme and stays true in every selector state. The title now carries the head term and the "(WSD)" disambiguator against the remote-sensing benchmark of the same name, and the meta description is generated from live counts so it cannot drift. Separately, nothing in CI inspected outbound links; the glite-lens 404 sat on the leaderboard undetected because it was a Python string rather than a markdown link. A lychee job now covers README, docs and baselines.py, with localhost and placeholder examples excluded. Co-authored-by: Claude Opus 5 (1M context) --- .agents/skills/write-homepage-answer/SKILL.md | 96 ++++++++++++++ .github/workflows/ci.yml | 18 +++ .github/workflows/pages.yml | 1 + src/sensebench/site/build.py | 16 ++- src/sensebench/site/templates/index.html.j2 | 27 ++++ tests/test_site.py | 38 ++++++ tools/check_homepage_answer.py | 125 ++++++++++++++++++ 7 files changed, 319 insertions(+), 2 deletions(-) create mode 100644 .agents/skills/write-homepage-answer/SKILL.md create mode 100644 tools/check_homepage_answer.py diff --git a/.agents/skills/write-homepage-answer/SKILL.md b/.agents/skills/write-homepage-answer/SKILL.md new file mode 100644 index 0000000..416a772 --- /dev/null +++ b/.agents/skills/write-homepage-answer/SKILL.md @@ -0,0 +1,96 @@ +--- +name: "write-homepage-answer" +description: "Rewrite the hand-written answer block on the SenseBench homepage from current leaderboard data. Use when tools/check_homepage_answer.py reports it stale, or after runs that change the top of the board." +--- +# Write the Homepage Answer Block + +**Version**: 1 + +## Goal + +Rewrite the `
` in `src/sensebench/site/templates/index.html.j2` so +that it answers "which LLM is best at word sense disambiguation" truthfully from current data. + +This block is deliberately hand-written rather than template-generated. Its interesting sentences — +which models are statistically tied, what separates ConSeC from LENS — are judgements a format +string cannot make. That is also why it can go stale, and why every figure in it is machine-checked. + +## Inputs + +* `$ARGUMENTS` — optional. A results directory; defaults to `results/`. + +## Context + +Read before starting: + +* `src/sensebench/site/templates/index.html.j2` — the current block, and the `.intro` / `.note` + classes it reuses +* `src/sensebench/leaderboard/baselines.py` — baseline labels and, critically, the `source_note` + provenance caveats +* `src/sensebench/leaderboard/schemes.py` — the nine schemes and `DEFAULT_SCHEME_ID` +* `tools/check_homepage_answer.py` — the checks the result must pass + +## Steps + +1. Rebuild the data. Do not read figures from `_site/` without rebuilding it first; a checked-out + `_site/` can be months stale and may predate whole baselines. + + ```bash + uv run sensebench site build --results-dir results --output-dir _site --strict + ``` + +2. Read `_site/data/leaderboard.json`. Take the top entry's accuracy, both confidence bounds, + `correct_count`, `item_count`, `display_label`, `reasoning_effort` and `prompt_id`; the baseline + accuracies; and `summary.verified_run_count` / `summary.model_count`. + +3. **Compute the statistical ties.** Load `results//predictions.jsonl` for the top run and + for every run within a few points of it, and run an exact two-sided McNemar test over the + discordant pairs. Report which models are indistinguishable from the leader at p >= 0.05, and + name the notable models that are *not*. Accuracy proximity is not a tie — runs 0.4 points apart + have tested both ways. + +4. Take the latest run date as `max(created_at)` across entries, for both the prose and the + `
+{# + Hand-written by an agent, not generated: the statistical-tie set and the ConSeC/LENS + training-data distinction are judgements a format string cannot make. Every figure is + guarded by test_homepage_answer_block_figures_match_the_leaderboard in tests/test_site.py, + so stale numbers fail CI. Rewrite with .agents/skills/write-homepage-answer. +#} +
+

Which LLM is best at word sense disambiguation?

+

+ As of , the best verified result on + lexEN v1 is + 95.60% (95% CI 95.00–96.17), from + GPT-5.5 at xhigh + reasoning effort under registered prompt p001 — + 4,647 of 4,861 polysemous English items. Only Claude Fable 5 is statistically + indistinguishable from it (95.21%, McNemar p = 0.20); Gemini 3.1 Pro, GPT-5.6 and Claude + Opus 5 all fall significantly below. WordNet's most-frequent-sense heuristic scores 61.55% + on the same items; among supervised systems ConSeC, trained on the original human labels, + reaches 84.88%, while Glite's own LENS, retrained on model-relabelled SemCor, reaches + 89.69%. Figures use the default labels — lexEN v1 gold at + WordNet fine granularity; + coarser sense inventories score substantially + higher. Every number is recomputed in CI from the stored raw API responses. +

+

198 verified runs · 63 models · latest run 25 July 2026

+
+