Answer "which model is best" on the homepage and check outbound links in CI - #218
Conversation
… in CI Nothing on the site answered the question people actually ask. The top accuracy existed only as an unattributed stat tile that a reader -- or a retrieval system -- had to join to a table row across 200 KB of HTML, and the homepage title was the generic "SenseBench Leaderboard". The homepage now opens the leaderboard with an H2 phrased as the question and one self-contained paragraph answering it: the top verified result with its confidence interval, the run and prompt that produced it, which models are statistically tied with it, and where the MFS, ConSeC and LENS baselines sit. Five links carry a reader from each claim to the artifact behind it. The paragraph is written by hand rather than generated. Its interesting sentences are judgements a format string cannot make: which models tie is an exact McNemar test over stored per-item predictions, not accuracy proximity -- runs 0.4 points apart have tested both ways -- and the gap between ConSeC and LENS is a training-data difference worth explaining rather than a ranking worth asserting. Only Claude Fable 5 currently ties with GPT-5.5 at p = 0.20; the paper's "top three families are statistically indistinguishable" no longer holds on current data. Hand-written prose goes stale, so tools/check_homepage_answer.py verifies every figure in the block against the built site data and fails when one drifts, and .agents/skills/write-homepage-answer records how to rewrite it -- including the traps: recompute rather than copy, always name the label scheme, and never call a Glite artifact "strongest" without the qualifier. Accuracy here is meaningless without its scheme. The same run scores 95.60% under lexEN fine and 85.39% under Raganato fine, and the top model changes with it, so the prose names its scheme and stays true in every selector state. The title now carries the head term and the "(WSD)" disambiguator against the remote-sensing benchmark of the same name, and the meta description is generated from live counts so it cannot drift. Separately, nothing in CI inspected outbound links; the glite-lens 404 sat on the leaderboard undetected because it was a Python string rather than a markdown link. A lychee job now covers README, docs and baselines.py, with localhost and placeholder examples excluded. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe homepage now renders dynamic leaderboard metadata, a title update, and a verified answer block. A new validator compares answer-block claims with generated leaderboard data and the latest run date. Homepage tests cover rendered content and the default scheme. Pages CI runs the validator, CI adds link checking, and a skill document defines the regeneration workflow. Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
tests/test_site.py (1)
771-774: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert the scheme qualifier in rendered HTML.
This only compares a constant; the answer block can omit or misname
lexEN v1/ WordNet fine-grained scoring and the test still passes. Assert the qualifier in the builtindex_html, or make the homepage checker enforce it.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e864d618-844b-44db-8e6c-f08f55bbc710
📒 Files selected for processing (7)
.agents/skills/write-homepage-answer/SKILL.md.github/workflows/ci.yml.github/workflows/pages.ymlsrc/sensebench/site/build.pysrc/sensebench/site/templates/index.html.j2tests/test_site.pytools/check_homepage_answer.py
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: build
🧰 Additional context used
🪛 ast-grep (0.44.1)
tools/check_homepage_answer.py
[warning] 63-63: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: PERCENT_PATTERN.findall(block)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
[warning] 90-90: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: TIME_PATTERN.findall(block)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
🪛 LanguageTool
.agents/skills/write-homepage-answer/SKILL.md
[style] ~93-~93: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...and never link the same target twice. * NEVER rewrite the rest of the page "for AI re...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🪛 zizmor (1.26.1)
.github/workflows/ci.yml
[warning] 25-27: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 22-38: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block
(excessive-permissions)
🔇 Additional comments (6)
src/sensebench/site/build.py (1)
1659-1660: LGTM!src/sensebench/site/templates/index.html.j2 (1)
42-60: LGTM!tests/test_site.py (2)
23-23: LGTM!Also applies to: 171-173, 175-175
745-768: LGTM!tools/check_homepage_answer.py (1)
102-125: LGTM!.github/workflows/pages.yml (1)
27-27: LGTM!
| links: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| # Scoped to the files that carry outbound links we control. The Glite LENS 404 lived | ||
| # in a Python string, not a markdown link, so baselines.py is included deliberately. | ||
| - uses: lycheeverse/lychee-action@v2 | ||
| with: | ||
| # --exclude-all-private drops the localhost vLLM example; the second exclude drops | ||
| # the literal `http://...` placeholder in the self-hosted docs. | ||
| args: >- | ||
| --no-progress --max-retries 3 --exclude-all-private | ||
| --exclude '^http://\.\.\.' | ||
| README.md docs src/sensebench/leaderboard/baselines.py | ||
| fail: true | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- ci.yml ---'
nl -ba .github/workflows/ci.yml | sed -n '1,120p'
printf '\n%s\n' '--- permissions references ---'
rg -n "permissions:|persist-credentials|GITHUB_TOKEN|lychee-action" .github/workflows -nRepository: GliteTech/sensebench
Length of output: 213
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- ci.yml ---'
nl -ba .github/workflows/ci.yml | sed -n '1,160p'
printf '\n%s\n' '--- permissions references ---'
rg -n "permissions:|persist-credentials|GITHUB_TOKEN|lychee-action" .github/workflowsRepository: GliteTech/sensebench
Length of output: 213
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- .github/workflows/ci.yml ---'
sed -n '1,220p' .github/workflows/ci.yml
printf '\n%s\n' '--- workflow-wide token/checkout settings ---'
rg -n "permissions:|persist-credentials|GITHUB_TOKEN|actions/checkout@v4|lychee-action@v2" .github/workflows .githubRepository: GliteTech/sensebench
Length of output: 3183
Tighten token scope for the link check. lychee-action doesn’t need a writable token here, and actions/checkout keeps credentials in the repo by default. Add permissions: contents: read and persist-credentials: false.
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 25-27: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 22-38: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block
(excessive-permissions)
Source: Linters/SAST tools
| def _index_description(*, summary: SiteSummary) -> str: | ||
| dataset_labels = ", ".join( | ||
| _dataset_version_label(version) for version in summary.dataset_versions | ||
| ) | ||
| dataset_clause = "" if len(dataset_labels) == 0 else f" on {dataset_labels}" | ||
| return ( | ||
| "Verified leaderboard for English word sense disambiguation: " | ||
| f"{summary.verified_run_count} audited runs across {summary.model_count} models" | ||
| f"{dataset_clause}. Top accuracy {_format_percent(summary.top_accuracy)}." | ||
| ) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Pluralise the run label for singleton sites.
This always renders 1 audited runs; the new fixture currently locks that typo. Use singular/plural selection and update tests/test_site.py:174.
Proposed fix
def _index_description(*, summary: SiteSummary) -> str:
+ run_word = "run" if summary.verified_run_count == 1 else "runs"
dataset_labels = ", ".join(
_dataset_version_label(version) for version in summary.dataset_versions
)
dataset_clause = "" if len(dataset_labels) == 0 else f" on {dataset_labels}"
return (
"Verified leaderboard for English word sense disambiguation: "
- f"{summary.verified_run_count} audited runs across {summary.model_count} models"
+ f"{summary.verified_run_count} audited {run_word} across {summary.model_count} models"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def _index_description(*, summary: SiteSummary) -> str: | |
| dataset_labels = ", ".join( | |
| _dataset_version_label(version) for version in summary.dataset_versions | |
| ) | |
| dataset_clause = "" if len(dataset_labels) == 0 else f" on {dataset_labels}" | |
| return ( | |
| "Verified leaderboard for English word sense disambiguation: " | |
| f"{summary.verified_run_count} audited runs across {summary.model_count} models" | |
| f"{dataset_clause}. Top accuracy {_format_percent(summary.top_accuracy)}." | |
| ) | |
| def _index_description(*, summary: SiteSummary) -> str: | |
| run_word = "run" if summary.verified_run_count == 1 else "runs" | |
| dataset_labels = ", ".join( | |
| _dataset_version_label(version) for version in summary.dataset_versions | |
| ) | |
| dataset_clause = "" if len(dataset_labels) == 0 else f" on {dataset_labels}" | |
| return ( | |
| "Verified leaderboard for English word sense disambiguation: " | |
| f"{summary.verified_run_count} audited {run_word} across {summary.model_count} models" | |
| f"{dataset_clause}. Top accuracy {_format_percent(summary.top_accuracy)}." | |
| ) |
| BLOCK_PATTERN: re.Pattern[str] = re.compile( | ||
| r'<section class="answer-block".*?</section>', re.DOTALL | ||
| ) | ||
| PERCENT_PATTERN: re.Pattern[str] = re.compile(r"(\d+\.\d{2})%") | ||
| COUNT_PATTERN: re.Pattern[str] = re.compile(r"(\d{1,3}(?:,\d{3})+|\b\d{2,4}\b)") | ||
| TIME_PATTERN: re.Pattern[str] = re.compile(r'<time datetime="(\d{4}-\d{2}-\d{2})"') | ||
|
|
||
| REQUIRED_BASELINE_LABELS: tuple[str, ...] = ("MFS (WordNet first sense)", "ConSeC", "Glite LENS") | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class Failure: | ||
| claim: str | ||
| detail: str | ||
|
|
||
|
|
||
| def _pct(value: float) -> str: | ||
| return f"{value * 100:.2f}" | ||
|
|
||
|
|
||
| def _check(*, index_html: Path, leaderboard_json: Path) -> list[Failure]: | ||
| html_text = index_html.read_text(encoding="utf-8") | ||
| match = BLOCK_PATTERN.search(html_text) | ||
| if match is None: | ||
| return [Failure("answer block", f'no <section class="answer-block"> in {index_html}')] | ||
| block = match.group(0) | ||
|
|
||
| payload = json.loads(leaderboard_json.read_text(encoding="utf-8")) | ||
| entries = payload["entries"] | ||
| baselines = {row["label"]: row for row in payload.get("baselines", [])} | ||
| summary = payload["summary"] | ||
| top = entries[0] | ||
|
|
||
| failures: list[Failure] = [] | ||
|
|
||
| # Every figure the prose is allowed to state, derived from live data. | ||
| allowed = {_pct(entry["accuracy"]) for entry in entries if entry.get("accuracy") is not None} | ||
| allowed |= {_pct(row["accuracy"]) for row in baselines.values() if row.get("accuracy")} | ||
| allowed |= {_pct(top["accuracy_ci"]["low"]), _pct(top["accuracy_ci"]["high"])} | ||
|
|
||
| for stated in PERCENT_PATTERN.findall(block): | ||
| if stated not in allowed: | ||
| failures.append( | ||
| Failure(f"{stated}%", "matches no current run, baseline or confidence bound") | ||
| ) | ||
|
|
||
| def require(label: str, value: str) -> None: | ||
| if value not in block: | ||
| failures.append(Failure(label, f"expected {value!r}, not present in the block")) | ||
|
|
||
| require("top accuracy", f"{_pct(top['accuracy'])}%") | ||
| require("CI low", f"{_pct(top['accuracy_ci']['low'])}") | ||
| require("CI high", f"{_pct(top['accuracy_ci']['high'])}") | ||
| require("top model", top["display_label"] or top["model"]) | ||
| require("correct count", f"{top['correct_count']:,}") | ||
| require("item count", f"{top['item_count']:,}") | ||
| require("run count", str(summary["verified_run_count"])) | ||
| require("model count", str(summary["model_count"])) | ||
|
|
||
| for label in REQUIRED_BASELINE_LABELS: | ||
| row = baselines.get(label) | ||
| if row is None: | ||
| failures.append(Failure(f"baseline {label}", "absent from the built site data")) | ||
| continue | ||
| require(f"baseline {label}", f"{_pct(row['accuracy'])}%") | ||
|
|
||
| latest_run_date = max(entry["created_at"] for entry in entries)[:10] | ||
| stated_dates = TIME_PATTERN.findall(block) | ||
| if len(stated_dates) == 0: | ||
| failures.append(Failure("<time>", "the block states no machine-readable date")) | ||
| elif stated_dates[0] != latest_run_date: | ||
| failures.append( | ||
| Failure("<time>", f"states {stated_dates[0]}, latest run is {latest_run_date}") | ||
| ) | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Complete the homepage-integrity contract.
The checker does not validate several handwritten claims, so the skill’s “every figure is machine-checked” guarantee is currently false.
tools/check_homepage_answer.py#L24-L98: validate exact label/value associations, statistical comparisons, scheme and top-run metadata, links, and both date representations..agents/skills/write-homepage-answer/SKILL.md#L14-L16: retain the guarantee only after those checks exist.tests/test_site.py#L771-L774: assert the scheme qualifier in rendered HTML rather than checking only the constant.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 63-63: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: PERCENT_PATTERN.findall(block)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
[warning] 90-90: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: TIME_PATTERN.findall(block)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
📍 Affects 3 files
tools/check_homepage_answer.py#L24-L98(this comment).agents/skills/write-homepage-answer/SKILL.md#L14-L16tests/test_site.py#L771-L774
1 participant
What
The homepage now opens with the question and answers it in one self-contained paragraph:
Five links carry a reader from each claim to the artifact behind it: the top run page, the registered prompt, the lexEN repo,
/label-schemes/,/coarsening/.Why it is hand-written, and how that is kept safe
The interesting sentences are judgements a template cannot make. Which models tie is an exact McNemar test over stored per-item predictions, not accuracy proximity — runs 0.4 points apart have tested both ways. Worth noting: the paper's "the top three families are statistically indistinguishable" no longer holds on current data. Only Claude Fable 5 ties (p = 0.20); Gemini 3.1 Pro (p = 0.031), GPT-5.6 (p = 0.019) and Claude Opus 5 (p = 0.004) all differ.
Hand-written prose drifts, so:
tools/check_homepage_answer.pyverifies every figure against built site data and fails on drift. Demonstrated: injecting a bogus accuracy and model name produces.agents/skills/write-homepage-answerrecords how to rewrite it, including the traps I hit while drafting — recompute rather than copy, always name the scheme, never call a Glite artifact "strongest" without the training-data qualifier.Scheme qualifier
Accuracy here is meaningless without its label scheme: the same run scores 95.60% under
lexen_fineand 85.39% underraganato_fine, and the top model changes with it. The prose names its scheme, so it stays true in every selector state.Also
SenseBench (WSD): LLM Word Sense Disambiguation Leaderboard, committing to the(WSD)house form against arXiv:2605.10576. Meta description generated from live counts.glite-lens404 survived. Covers README,docs/, andbaselines.py(the dead link was a Python string, not markdown). All 22 real URLs verified 200; localhost andhttp://...examples excluded.