Skip to content

evals: gate clip selection on the clips it produces - #1

Open
hrithiksaini99 wants to merge 2 commits into
mainfrom
eval/clip-selection-gate
Open

hrithiksaini99 wants to merge 2 commits into
mainfrom
eval/clip-selection-gate

Conversation

@hrithiksaini99

Copy link
Copy Markdown
Owner

What

Adds an eval gate over the real clip selector (src/studio/select.py), using agenteval.

The unit tests assert how a score is computed. They cannot see a weight change that silently drops the strongest hook, or a window that ends mid-sentence. This runs the real load_sentences → find_clips path over three committed transcript fixtures and asserts on the clips that come out.

Coverage

Area Cases What would break it
Hook detection hook-heavy-selects-both-hooks A scoring change that stops surfacing the strongest openers
Score floor hook-heavy-scores-above-floor Weights drifting down until any span looks acceptable
Duration bounds hook-heavy-structure, …-respects-duration-bounds An off-by-one in the 28–75s window
Non-overlap hook-heavy-structure, filler-heavy-structure Greedy selection losing its min_gap_seconds check
Complete thoughts hook-heavy-selects-both-hooks Windows ending mid-sentence
Parameters …-respects-max-clips, …-respects-duration-bounds max_clips / max_seconds no longer plumbed through
Degenerate input short-episode-yields-nothing A 14-second episode producing a 28-second clip
8/8 passed (100.0%)

Verified against a real regression

I disabled hook weighting (hook * 1.7 → hook * 0.0) and reran against the committed baseline:

7/8 passed (87.5%)
gate: regressed against baseline: hook-heavy-scores-above-floor
  FAIL hook-heavy-scores-above-floor: below floor [(6.41, 'Welcome back...'), (7.22, 'Seven of those...')]
exit=1

Restored the file; the suite is green again. A selector change that no unit test would flag now fails with a case id instead of a slightly different pass rate.

Findings from the first run

  1. There is no minimum score. find_clips returns the best windows regardless of absolute quality, so a 100-second episode of pure hedging still produces two clips (4.10 and 6.56, against 10.78 and 11.27 for the real episode). Recorded as "optional": true so it stays visible in every report without blocking merges. A min_score parameter is the fix.
  2. Results are ordered by start time, not score. Correct product behaviour — shortlists should read in episode order — but easy to assume otherwise when reading the code. clips_ordered now pins it.

Files

evals/adapter.py           the only file that knows both sides
evals/suite.jsonl          8 cases, one per line
evals/baseline.json        committed reference run
evals/build_fixtures.py    fixtures as readable prose, regenerable
evals/fixtures/*.json      3 transcripts, one segment per line
evals/README.md            coverage, findings, how to bump the baseline

Reviewer notes

  • evals/baseline.json is the reference run. Bump it only for intentional behaviour changes, and say why in the commit — an unexplained baseline bump turns the gate into decoration.
  • The workflow file is not in this PR. The token that pushed this branch has repo but not the workflow scope, so GitHub rejected .github/workflows/eval-gate.yml. It is ready and will land as a one-file follow-up once gh auth refresh -s workflow is run. Until then the suite is run manually.
  • agenteval is not on PyPI yet. Local runs need pip install -e ../agent-eval. The follow-up workflow pins to a tag once the harness is published, so a change in the harness cannot silently change this repo's gate.

Test plan

pip install -e ../agent-eval
PYTHONPATH=src python -m agenteval run --suite evals/suite.jsonl \
  --agent evals/adapter.py:run --baseline evals/baseline.json \
  --report evals/report.md --fail-on-regression

Unit tests assert how the score is computed. They cannot see a weight change
that silently drops the strongest hook, or a window that ends mid-sentence.

Adds 8 cases over 3 committed transcript fixtures, run through the real
load_sentences -> find_clips path:

- duration bounds, non-overlap, playback order, sentence-complete endings
- the strongest early and late hooks still surface
- max_clips / max_seconds plumbing
- a 14s episode yields no clips

A committed baseline means a selector regression fails CI with the case id
rather than a slightly different pass rate.

The first run found two things: find_clips has no minimum score, so a hedging
100s episode still yields two clips; and results are ordered by start time, not
by score. The former is recorded as an optional case so it stays visible in
every report without blocking merges until there is a fix.
Unrelated to the eval suite, but it blocks verifying it: CI started resolving
SQLAlchemy 2.1.6, which no longer ships sqlalchemy.ext.mypy.plugin, so
'pytest && ruff' pass and 'mypy src' stops at the plugin import.

Pin to the 2.0 line until the mypy plugin situation is handled deliberately
(remove the plugin from pyproject, or move the models off it).

Last green CI on main was 2026-09-03, before 2.1 was released.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant