feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced by uipreliga · Pull Request #109 · UiPath/coder_eval

uipreliga · 2026-08-12T21:36:21Z

What

Adds Dataset.split_field + a --split row filter, ships a seventh plugin skill /coder-eval:optimize-skill, and dogfoods the whole loop in a new tutorial — which is where most of the value in this PR came from.

Implements c/2026-08-12-skill-optimize-p0-p1.md, phase by phase.

The feature

--split (P0). Label dataset rows train / test and select one at run time. The filter runs before either sampler — sampling first would leave an unpredictable (possibly zero) number of rows per split, destroying the comparison the split exists to protect. Unlabelled tasks pass through untouched, so --split is safe in a multi-task run; a labelled task with no matching row raises.

/coder-eval:optimize-skill (P1). A/B tests candidate edits to a skill as experiment variants, promoting only what beats run-to-run noise and then survives a held-out split. Two tracks:

Activation — the frontmatter description, measured against an activation suite. Does it fire when it should?
Execution — the skill body, measured against an outcome suite with real success criteria. Having fired, does it do the job?

Explicit-invocation only — it spends real money.

Why the two gates differ (worth a reviewer's eye)

The tracks share splits, snapshots, reachability, replicate discipline and the ledger. What differs is the instrument, and it is load-bearing:

Activation compares F1, which a pooled suite.json cannot report per replicate — hence three separate invocations.
Execution compares per-row weighted_score, which paired_comparison already computes correctly over replicates it averages per row before pairing — hence --repeats 3 on exactly two variants.

So the paired block that is mere corroboration on the activation track is the primary instrument on the execution track: tested code instead of arithmetic by hand. Unifying the gates would swap in an instrument that cannot see the metric, so both halves are pinned by sensors.

Two smaller inversions, also sensored: activation rows must never name the skill (that tests obedience, not activation) while execution rows must invoke it by slash command — /plugin:skill at the top of initial_prompt — to hold activation constant; and disable-model-invocation: true now routes to the execution track rather than hard-stopping, so init and ci are optimizable after all.

The slash form is not a stylistic preference. A disable-model-invocation skill is not offered to the model at all, so asking in prose returns "no such skill is available" and the row measures nothing; the slash form loads it, emits a real Skill tool call, and is detected by skill_triggered. Both halves verified live (see below).

Validation

Beyond the test suite, the skill itself was live-tested in three layers, because most of its failure modes are silent rather than loud:

Static — every ${CLAUDE_PLUGIN_ROOT} reference, slash command and CLI flag it emits resolves; the experiment YAML in its body validates against ExperimentDefinition.
Cold agents — two agents with no context executed the skill end to end, one per track, on a zero-run budget, building real snapshots and experiment YAML for inspection.
Live probe — one real agent run to settle whether skill_triggered detects a slash-command invocation (it does; scored 1.0).

That surfaced 15 problems, fixed in 9ac72f6. The two worth a reviewer's attention were silent confounds in the snapshot step: the diagram showed only skills/ while the prose said "everything that source contained" (a diagram-faithful snapshot mounts skills whose bundled reference/ files are missing — invisible on the activation track), and omitting .claude-plugin/plugin.json makes the namespace default to the arm's directory name, so arms would differ in the listing name as well as the text under test. Neither would have errored; both would have produced confident, wrong numbers.

It also produced one piece of guidance now in the skill: before proposing an A/B, check whether a lint rule already answers the question for zero runs. The complaint that prompted the ci test — "emits workflows that miss a step" — overlaps CE026 almost exactly, and ~300 agent runs is the wrong way to learn something a static rule catches permanently.

Three bugs caught before shipping

Two were errors in the plan, found by review; the third was already shipped on main.

The sibling-regression gate read the wrong metric. Annexation makes the sibling's criterion expected=yes, observed=no — a false negative. Since precision = tp/(tp+fp), it stays pinned at 1.0 however many requests are stolen. The gate would have been gating on a constant. Now reads recall.yes.
A one-skill candidate snapshot empties the sandbox. A variant's plugins block replaces the task's, so the snapshot is the arm's only skill source. Snapshot one skill and every sibling criterion silently observes no — the sibling gate "passes" by measuring nothing. Snapshots now carry the whole skills tree.
The shipped reachability guidance was wrong (pre-existing). A local plugin path must be a plugin root holding skills/; the template told users to point at the bare skills directory, which loads nothing. Proven by probing three layouts against the real CLI. Every suite check-skill generated would have reported recall 0.0 — exactly what the template's own comment calls "reads exactly like a broken skill". Corrected in 7 surfaces, including ci, which was writing the broken path into users' CI workflows.

The tutorial (docs/tutorials/08)

Runs the loop for real against this repo's own skills, on Sonnet, and reports what happened rather than a tidied version:

lint-tasks measured at ceiling (F1 1.000 both splits), so the loop declined to spend ~224 runs chasing an unreachable number. That also closed the plan's open residual: the earlier 66-character description trim is now measured safe.
A task misfire that reproduced on both splits turned out to be 2-in-3 variance under replicates. Two agreeing runs were not evidence.
The real headroom was a sibling: analyze missed "what regressed" deterministically. Full three-stage A/B → a-regression promoted, gated on the train split (1.000 vs 0.667, non-overlapping, three invocations) and confirmed on the test split (1.000 vs 0.909).
Two comparisons were invalidated mid-run and thrown out — one candidate ranked on an eroded denominator, and a confident p = 0.038 that was a billing limit eating one arm harder than the other. completion_rate is what caught both.
Bare-name collision hazard: skill_triggered strips plugin: prefixes, and Claude Code ships its own unscoped init. A skill_name: "init" criterion would silently score a different skill. Documented in check-skill.

Notes for review

Listing budget held flat at 1,574/1,600 with a seventh skill added — five descriptions trimmed rather than raising the ceiling, since the budget is shared with every skill the user has installed.
analyze's description change is the one behavioural edit to an existing skill, and it is measured.
3 new lint sensors (mutation-tested); 1 harness gap deferred to .claude/harness-candidates.md — an all-skipped run exits 0, now reachable by a one-character --split typo. Fixing it changes exit semantics for every skipped-task path, so it wants a decision rather than a drive-by.

make verify green: 4071 passed, coverage 91.57%.

🤖 Generated with Claude Code

Late changes (after the first review pass)

Two follow-ups from using the thing, both pushed since the PR opened:

Split values renamed tune/holdout → train/test. Docs and content only — split values are open strings, so no schema change and nothing to migrate; split_field still defaults to "split". 215 replacements, followed by a pass disambiguating bare "test" back to "test split" wherever it named the data half rather than a check, since in a testing framework "without a test" parses exactly wrong. (Pedantic but recorded: nothing is trained here — you hand-select a candidate, which is a form of fitting, so "train" is defensible in the ML sense and far more recognisable than "tune".)

The skill now labels the splits itself rather than asking the user to. It previously offered to add labels, leaving a mechanical JSONL edit and a balance judgement to the reader. It now writes them and shows the counts for objection, under two rules that are easy to get wrong by eye and expensive when you do: stratify within each polarity so both halves carry positives and distractors, and assign deterministically so the split never re-rolls between rounds — a reshuffled split is not a test split, because rows already tuned against leak into it.

Not done, deliberately: a hash-based auto-split (--split train --split-ratio 0.6, membership derived from row_id, no labels in the file) would remove the last of the friction, but it conflicts with a load-bearing behaviour — unlabelled tasks currently pass through unfiltered, which is what makes --split safe in a multi-task run. Auto-splitting them would silently change that, so it needs an opt-in flag and its own tests. Left as a follow-up rather than smuggled in here.

Late changes (second pass): the execution track measured for real — and a criterion bug it uncovered

The execution track had never been run end to end. Doing that turned up a bug in skill_triggered that invalidates two claims made earlier in this description, so those are corrected below rather than quietly edited.

⚠️ Behaviour change reviewers should look at

skill_triggered no longer counts an errored Skill call as engagement (src/coder_eval/criteria/skill_triggered.py).

Any suite where a Skill call failed previously scored yes and will now score no. That is the correction — a refused call means the skill's body never loaded — but an existing green activation suite that was passing on refused calls will go red, and correctly so. The file-read signal is deliberately not gated the same way: a refused call loaded nothing, whereas a path reference means the SKILL.md was actually opened. Both cases are pinned by tests.

What the run found

An outcome suite was built to A/B ci's body: 4 arms differing only in that body, 24 rows. All four tied exactly on every criterion — which is a bug report, not a result.

ci sets disable-model-invocation: true, and the Skill tool refuses such a call outright:

<tool_use_error>Skill coder-eval:ci cannot be used with Skill tool
due to disable-model-invocation</tool_use_error>

24 of 24 calls failed this way, result_status: "error" on every one, and no row read the SKILL.md off disk either. The body never entered context; the agent answered from its own knowledge of GitHub Actions, plausibly enough that nothing downstream looked wrong. The tell was in the output the whole time:

body NOT loaded:  uses: anthropics/coder-eval-action@v1   <- does not exist
body loaded:    - uses: UiPath/coder_eval@v0             <- what the body specifies

Re-scoring those same artifacts with the fixed criterion gives 0/6 engagement per arm where the run reported 6/6.

Corrections to claims made above

"asking in prose returns no such skill is available" — wrong. Measured across the same six rows, by how often the model attempted the call: slash form alone 3/6, prose instruction alone 5/6, slash + explicit imperative 6/6. Prose works better than the slash form. Nothing in coder-eval expands a slash command; it arrives as plain text the model may ignore.
"Live probe — … whether skill_triggered detects a slash-command invocation (it does; scored 1.0)" — that 1.0 was the false positive. The call had errored.
"init and ci are optimizable after all" — true, but only with the snapshot fix: delete the disable-model-invocation: line in every arm's snapshot, which is what a real user's slash command effectively does. Verified — the same rows then score 1.000 with the action reference correct.

Also in this pass

Verdict on ci: no promotion, a verified ceiling. With the body loaded, the train split scores 1.000 on every row and every criterion (18/18 assertions, completion_rate 1.0). The three candidates were solving a problem that did not exist. Stages B and C were correctly never reached. ~55 Sonnet runs, ~$20.
The outcome-suite template (reference/templates/outcome.yaml) + a checked-in worked example (tasks/skills/ci-outcome.yaml), runnable in two documented lines.
Engagement is now actually gated (recall.yes: 1.0) in both — they described it as a gate and carried no threshold, which is how a zero-engagement run exited 0.
CE035 (partly-labelled split datasets) and CE036 (a row's prompt must not contain the string its criteria grade — four scenarios here had that leak).
Tutorial 09, written from the real numbers, plus tutorial 08 made reproducible (snapshot layout, per-stage experiment YAML, the -e command lines) and restructured into two parts.
docs/TASK_DEFINITION_GUIDE.md updated — it documented the pre-fix detection rule.

make verify green: 4074 passed, coverage 91.57%.

Third pass — the review's P0/P1/P2 fixes, then a tutorial audit

Implements c/2026-08-13-optimize-skill-review-v2-fixes.md, one commit per phase.

Two behaviour changes to look at first

skill_triggered now requires the signal to have actually delivered. The early-stop watcher evaluates on the tool call, where result_status is still None, so an in-flight or crash-force-closed Skill call live-passed while the frozen check scored it no — live and final verdicts disagreed on the one criterion that decides whether a paid round measured the skill at all. Engagement is now an allowlist: a Skill call counts only on "success" (for that tool the body is the tool result), and a Read/Glob/Grep that errored or has not resolved does not count — the path is in its parameters, but nothing loaded. Bash stays ungated, because cat … | grep exits non-zero after genuinely reading the file. Historical activation P/R/F1 is not directly comparable if a run contained those shapes; two golden-corpus entries were appended to force that acknowledgement rather than editing existing ones.

This reverses a documented design decision, so it is worth a reviewer's eye. Seven tests in tests/test_early_stop.py pinned the opposite semantics as deliberate; each is inverted, not deleted, with the reasoning in its comment, plus a Bash twin for the path that still decides on the call. The cost is stated in the test that pays it: a Skill call whose result never arrives no longer stops the run, so that run continues to its turn cap. The old behaviour stopped it and scored yes — crediting a skill whose body never reached the agent.

A mistyped --split now exits 2 instead of 0. It used to raise, get caught like any load failure, and land in skipped_tasks: one yellow line, zero evaluations, exit 0 — a CI gate reporting success for a one-character typo. SplitSelectorError separates a malformed file (still demoted, so one bad task cannot abort a suite) from a malformed invocation (re-raised — the same selector applies to every task in the run). No new CLI code was needed: the existing except ValueError -> typer.BadParameter supplies the exit.

The rest

coder-eval plan expands datasets and takes --split — prints total/selected row counts and catches a bad ${row.*} at plan time rather than per-row after the sandbox is built. The pre-spend check now costs nothing. It is also louder: a task with a missing dataset JSONL now fails a plan it previously passed.
Partial split labelling stops being silent — a WARNING with the drop count from expand_dataset, a yellow ⚠ from plan, and a check in lint-tasks for adopters CE035 cannot reach. All three call row_split_label, so the rule cannot drift even though the messages differ.
Row-id validation covers the whole dataset, before --split and sampling narrow it. A malformed row in the test half used to validate under every --split train run and surface at promotion time — the most expensive moment to learn it.
The bundled activation.yaml caps and isolates (run_limits, setting_sources: []), mirroring the checked-in worked example key for key. It preached both and shipped neither, so a user who copied it got the opposite of the advice they had just read.
CE036 gets real fixtures and a skill_name exemption. Its only test was a repo scan that passed whether or not the rule could still detect anything; detection now lives in a helper the scan and six fixtures share. CE036_LOCATOR_FIELDS is the source and CLAUDE.md the derived surface, with a both-directions parity sensor — that list had already drifted once.
reference/optimize-method.md — the track-invariant method extracted from an 859-line SKILL.md as a byte-identical move (verified against HEAD), the three named seams fixed, and the prose sensor split into procedure-vs-method sets that state which file each token belongs to and why.
Method caveats the review asked for: promotion verdicts apply to the flag-removed configuration, and Stage B now requires a predeclared primary criterion plus guardrails — scanning every per-criterion aggregate post hoc is uncorrected multiple testing in the rejection direction, so noisy criteria veto real wins.
ci-outcome's both-triggers row asked for two triggers and graded one, so a schedule-only workflow scored 1.000 — and it graded the same string as its sibling, discriminating nothing. Fixed without touching any train row, so the file's own measured "the train split scores 1.000 on every row" claim stays true by construction.

Tutorial audit

All nine checked mechanically — every path, relative link, cross-doc anchor, CLI flag against live --help, and every complete YAML snippet against the real models. All clean. By content, three defects, every one the same shape: a hand-maintained number describing a set the code derives.

"14 criterion types" (×3) against a registry of 15. CLAUDE.md's heading was right, which is why nothing noticed — it was the only guarded surface.
"six commands" in tutorial 07, omitting optimize-skill entirely. A skill-count sensor already existed; this file was not one of the four surfaces it read, and the tutorial's phrasing ("commands") was a fourth the matcher did not know.
Tutorial 09 quoting ci-outcome.yaml excerpts this PR had just changed — a reader copying the shown row would have built a suite that raises at expansion.

Each is now derived rather than asserted, and tutorial 07 joins SKILL_DOC_SURFACES.

Tutorial 08 Part 1 was re-measured rather than annotated — 62 Sonnet runs, three replicates of the train split plus one of the test split. lint-tasks still holds at F1 1.000 on a suite three rows larger. analyze reads 1.000 where Part 1 recorded 0.000, because the description this PR promotes is committed (4c7481c): the baseline is unreproducible precisely because the loop worked. And hard-3 is still unstable at exactly the 2-in-3 Step 6 measured — months later, on a different revision of the suite, which is the page's own "two agreeing runs are not evidence" lesson arriving unprompted. The page now states what a reader gets today and why it differs, instead of quietly refreshing the numbers.

Notes for review

6 new sensors + 6 CE036 fixtures, all mutation-verified — each was confirmed to fail when the property it guards is reverted.
1 harness gap deferred: "the ToolStart seam decides" is now a per-criterion property rather than a global invariant. Not mechanically detectable — it is about what a live_verdict reads, which no AST rule can infer.
No new Pydantic model, criterion, agent, config key or MergeField; one new exception type (SplitSelectorError), one module logger, one CLI option on an existing command.

make verify green: 4115 passed, 390 lint checks, coverage 91.63%.

Label dataset rows with a split (tune / holdout / …) and select one at run time with `coder-eval run --split <name>`. The filter runs BEFORE either sampler: sampling first would leave an unpredictable (possibly zero) number of rows per split, destroying the comparison the split exists to protect. - `Dataset.split_field` (default "split") mirrors `stratify_field`'s shape. - The filter is inlined in `expand_dataset` rather than extracted: one call site, a one-line comprehension, and a helper would re-declare the missing-field convention `_stratified_sample` already owns. - A row is unlabelled when the field is absent, null, or "". A task whose rows are all unlabelled passes through unfiltered — `--split` is global to the invocation, so an unlabelled suite beside a labelled one must not fail. A labelled task with no matching row raises, naming the splits that exist; `resolve_all_tasks` records that as a skipped task, so a mistyped selector is a zero-task run that still exits 0. Documented rather than papered over. `--split` unset leaves expansion byte-for-byte unchanged (pinned by test). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…activation template `/coder-eval:optimize-skill` turns an activation suite's confusion matrix into candidate description rewrites, A/B tests them as experiment variants, and promotes only what beats run-to-run noise and then survives a held-out split. Explicit-invocation only: it spends real money across three stages. Two mismeasurements were designed out rather than discovered later: - The sibling-regression gate reads the sibling's `recall.yes`, not its precision. Annexation makes the sibling's criterion expected=yes/observed=no — a false negative — and `precision = tp/(tp+fp)` stays pinned at 1.0 when the sibling never misfires, so a precision gate would gate on a constant. - Each candidate snapshots the WHOLE skills directory, siblings copied unchanged. A variant's `plugins` block replaces the task's, so the snapshot is the arm's only skill source: snapshot one skill and every sibling criterion silently observes `no` in every arm, and the description is tested against a listing it will never face. Supporting changes: the activation template gains `split_field` and per-row tune/holdout labels (both splits carry positives and distractors; no `stop_early:` — that would degrade sibling measurement); run-layout.md documents the suite-rollup path, aggregate shape, `failed_samples` as the only row-identity field, and replicate pooling — the contract that keeps someone from "simplifying" Stage B's three invocations into `--repeats 3`, which pools into one suite.json and leaves the gate nothing to read. Five shipped descriptions trimmed so seven skills fit the listing budget (1,524/1,600) without raising the ceiling: the budget is shared with every skill the user has installed, so growing our own footprint evicts theirs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…lity guidance it disproved Runs the optimization loop end-to-end against this repo's own `lint-tasks` skill and reports what actually happened, including the parts that did not go to plan. The shipped reachability guidance was wrong. Both the activation template and `check-skill` said a local plugin `path` is "the directory containing the skill's directory" — `.claude/skills`. Probing three layouts against the real CLI shows a plugin path must be a PLUGIN ROOT holding a `skills/` subdirectory (`<path>/skills/<name>/SKILL.md`; a manifest is optional, the namespace then defaults to the directory name). A bare directory of skill directories loads nothing at all. So the correct root for `.claude/skills/my-skill/SKILL.md` is `.claude`. Every suite `check-skill` generated would have reported recall 0.0 — precisely the symptom the template's own comment warns "reads exactly like a broken skill". Corrected in the template, `check-skill` and `optimize-skill`, and pinned by a new sensor. The run itself: `lint-tasks` scores F1 1.000 on both splits, so the loop stops at the diagnosis step rather than spending ~130 further runs chasing a number the gate makes unreachable. That result closes an open question from the previous commit — the 66-character description trim is now measured safe rather than assumed. The headroom turned out to be in the sibling matrix: `task` annexes setup requests on both splits, and `analyze` never fires on "what regressed". The tutorial ends on that, because a walkthrough that only works when the result is positive teaches the wrong lesson about A/B testing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ing --split Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The reachability correction reached only half its surfaces. `ci`'s workflow snippet still exported `SKILL_SOURCE_PATH=.../.claude/skills` — a shipped skill writing the broken path straight into users' CI, where the failure mode is a permanent red that looks exactly like the drift the schedule exists to detect. Same stale form in docs/PLUGIN.md and tutorial 07, plus a self-contradicting sentence in optimize-skill's own Step 8. All corrected, and the sensor now covers seven surfaces and asserts the specific layout: its previous `"skills/" in text` check passed on the pre-fix text via `.claude/skills/my-skill/SKILL.md`, so it guarded nothing. `check-skill` was never told about splits, so following it to the documented row count produced a PARTLY labelled suite — the one state where `--split` silently drops rows and shrinks the denominator the thresholds gate on. It now labels every row it writes, and optimize-skill gained the missing branch. Two rows of the new repo suite asserted `expected_skill: "init"`, a skill that sets `disable-model-invocation: true` and so can never be engaged: an unsatisfiable criterion that made every run of that suite exit 1. Relabelled to `""`, which asks the question that has an answer. Also: duplicate row ids are now rejected across the whole dataset before `--split` narrows it, so a duplicate in an unselected split can no longer validate under every split and surface only on a full run; run-layout documents `rows_total` / `rows_excluded` / `completion_rate`; and the claim that an excluded row shrinks a suite "silently" was wrong — the rollup reports it and it is gateable. Re-ran both splits after the label fix. `lint-tasks` holds F1 1.000. The `task`-annexes-setup finding, which had reproduced on both splits, did NOT survive the third run on byte-identical prompts — so the tutorial now teaches that instead, which is a better lesson about why the gate demands replicates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… two open findings Both follow-ups from the previous round are now settled by measurement rather than left as open questions. **The `task` misfire was intermittent, not phantom.** Three tune invocations show `hard-3` engaging `task` in two of three runs on byte-identical prompts. The earlier pair of agreeing runs was luck. This is the clearest argument yet for the replicate gate, and the tutorial now teaches it with the real trace. **The `analyze` gap was real, stable, and fixable.** Recall 0.500 in all three runs with precision 1.000 throughout — one row ("what regressed") missed every time, no over-claiming. The suite gained analyze rows on both splits so the gap could be measured at all, then the full loop ran: - Stage A (68 runs): three candidates, all above the incumbent. `b-results` looked competitive but lost a row to an error (completion 0.941), so its recall came from 3 rows not 4 — ranked out rather than compared. - Stage B (153 runs, three separate invocations): `a-regression` at F1 1.000 in every run against a rock-steady incumbent 0.667. Non-overlapping, no sibling regression, precision never off 1.000. One incumbent invocation dropped a row and was excluded rather than averaged in. - Stage C: not obtained. The first holdout was uninformative — a flat tie, because every regression-phrased row had been put in tune. Fresh holdout rows were authored and the re-run then hit the org's monthly spend limit, which removed 11 rows from one arm and 6 from the other and produced a confident p = 0.038 pointing the WRONG way. A p-value over an asymmetrically eroded sample is not evidence. `a-regression` is therefore promoted **gated on tune, unconfirmed on holdout** — stated plainly in the tutorial rather than dressed up. Listing budget holds at 1,574/1,600. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…d on holdout Re-ran the holdout confirmation now that budget is available. Erosion was one row against the candidate and none against the incumbent — near-symmetric, and pointing the conservative way — so unlike the spend-limited attempt this run is interpretable. incumbent analyze recall 0.833 precision 1.000 F1 0.909 a-regression analyze recall 1.000 precision 1.000 F1 1.000 The direction reproduces on rows the candidate was never tuned against, which is what Stage C is required to show. One row separates the arms, and it is one of the fresh rows authored at promotion time: an-6 "Which of my tasks got worse after I switched the model?" incumbent 1 of 3 a-regression 3 of 3 No sibling regressed in either arm; the incumbent shows the intermittent `task` misfire once more, consistent with the 2-in-3 rate measured earlier. The paired comparison reads exactly zero (p = 1.000). That is the documented limit of the block rather than a contradiction: it pairs per-row weighted_score across all three criteria, so a gain confined to one criterion on one row out of eleven is diluted below what 11 pairs resolve. F1 is the promotion metric; the tutorial now reports both and says which one the decision rested on. `a-regression` therefore ships gated AND confirmed, and the earlier "unconfirmed" framing is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The Stage C per-row data showed `init` being engaged on the setup rows in both arms — which contradicted a comment claiming coder-eval's `init` "can never be engaged" because it sets `disable-model-invocation: true`. Both halves were wrong in an instructive way. Claude Code ships its own unscoped `init` skill, and `skill_triggered` matches on the BARE name (it strips any `plugin:` prefix), so the two are indistinguishable to the criterion. A `skill_name: "init"` criterion would have silently scored a different skill's activation as though it were the plugin's. That is a second, independent reason not to have written one — beyond the unsatisfiability already fixed. Documented where suite authors will hit it: `check-skill` (which already covers bare-name matching), the tutorial's caveat section, and the suite's own comment. A collision does not error; it measures the wrong skill. Also marks the plan file complete and ticks its 52 acceptance criteria, with a header recording the four deviations the implementation required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-08-12T21:37:14Z

Claude finished @uipreliga's task in 1m 29s —— View job

Code Review in Progress

I'm reviewing PR #109 following the guidelines in .github/code_review.md.

Todo List

Read .github/code_review.md for review standards
Read CLAUDE.md for project conventions
Run git diff origin/main...HEAD to see full changes
Review changed files with full context
Perform cross-file consistency checks
Analyze what's missing
Provide comprehensive review feedback

Starting review now...

… body, not just the description A skill can fail two independent ways: it never gets reached, or it gets reached and gives bad instructions. optimize-skill only measured the first. Both tracks now live in the one skill, because they share almost everything — splits, snapshots, reachability, replicate discipline, the ledger, the stop rule. What differs is the instrument, and that difference is load-bearing: `skill_triggered` is a binary one-turn probe that says nothing about the quality of the work that follows, so an activation suite cannot grade a body. The execution track uses an ordinary outcome suite with real success criteria. Three consequences worth calling out, each pinned by a sensor: - **The two gates differ on purpose.** Activation compares F1, which a pooled suite.json cannot report per replicate — hence three separate invocations. Execution compares per-row weighted_score, which `paired_comparison` already computes correctly over replicates it averages per row before pairing. So the paired block that is mere corroboration on the activation track is the PRIMARY instrument on the execution track: tested code instead of hand arithmetic. Unifying the gates would swap in an instrument that cannot see the metric. - **The prompt rule inverts.** Activation rows must never name the skill (that tests obedience, not activation). Execution rows must, to hold activation constant so the body is the only variable. - **`disable-model-invocation: true` now routes instead of stopping.** Such a skill's description never enters the activation decision, but its body still determines whether it does its job — so `init` and `ci` are optimizable after all, just not on the track that was previously the only one. Also: one variable per round (never both tracks at once), regression coverage for rows that already pass (a body edit breaks things silently, where the confusion matrix shows activation regressions for free), and trajectory-based diagnosis with the five failure modes a body edit actually addresses. Tutorial 08 now says which track it walks. Listing budget holds at 1,577/1,600. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two agents executed the skill with no prior context and a zero-run budget, one per track, building real artifacts. Between them they found 15 problems. The substantive ones: **The execution track never said how to engage the skill it routes to.** Step 2 sends every `disable-model-invocation` skill there, and Step 4 then said "name the skill in the prompt" — which does not work: such a skill is not offered to the model at all, so prose gets "no such skill is available" and the row measures nothing. Verified live, and verified the fix: a slash command in `initial_prompt` DOES load it, emits a real `Skill` tool call, and is detected by `skill_triggered` (probe scored 1.0). The routing was right; the mechanism was missing. Now specified, with a worked `initial_prompt`. **Two silent confounds in the snapshot step.** The diagram showed only `skills/` while the prose said "everything that source contained" — a diagram-faithful snapshot mounts skills whose `${CLAUDE_PLUGIN_ROOT}/reference/` files are gone, invisibly on the activation track. And dropping `.claude-plugin/plugin.json` makes the namespace default to the arm's directory name, so arms differ in the name shown in the listing as well as the text under test — on the one track where activation is a competition between listings. **A version string is not a capability check.** Two binaries reported the same 0.9.6 and differed in whether `--split` exists at all; the pinned-version rule said "carry on" while every run would fail at load. Step 1 now requires `coder-eval plan <suite>` to exit 0 before spending. **The skill writes descriptions and never mentioned the length ceiling** every natural fix pushes against. Now budgeted before candidates are written. Also: the baseline is now a line item in the cost table with its non-redundancy explained (it validates the snapshot wiring against the task's own source); `completion_rate` is documented as per-criterion-aggregate rather than top-level; the execution track's no-suite stop matches the activation track's force and carries the split-label and slash-invocation requirements into the hand-off; suite sizing resolves to a number rather than contradicting itself; Step 3 gains a "check whether a lint rule answers this for zero runs" pre-check; and repo-layout.md now prunes `tmp/` and any already-found run store, which had been returning hundreds of agent-produced SKILL.md files. Fixes two bugs this branch introduced: an off-by-one step reference from the renumbering, and tutorial 08 claiming the skill hard-stops where it now routes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…vention Tutorial 08 shipped without the YAML front-matter every other tutorial carries, and with an H1 that did not follow `# Tutorial NN — Title Case`. The front-matter is not decorative: mkdocs renders `description:` into the page's `<meta name="description">`, so the page had no search or social-preview summary at all. - adds the folded `description: >-` block, matching 01–07's shape and voice - retitles to `# Tutorial 08 — Optimizing a Skill Description` - title-cases the nav and tutorials/README entries, which were the only lowercase ones in either list - regenerates docs/llms.txt via `make docs-indexes` (verified idempotent) Audited the rest of the tree while here: 24 of 24 docs pages now carry a front-matter description; 08 was the only one missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…, and label them for the user Terminology: `train`/`test` is the vocabulary everyone already has, and split values are open strings, so this is docs and content only — no schema change, no migration. `Dataset.split_field` still defaults to `"split"`; only the values this project's own suites and guidance use have changed. 215 replacements across 19 files, then a pass to disambiguate bare "test" back to "test split" wherever it named the data half rather than a check — in a testing framework "without a test" reads as "without a unit test", which is exactly the wrong parse. One honest imprecision worth noting: nothing is *trained* here. You hand-select a description from candidates, which is a form of fitting, so "train" is defensible in the ML sense of "the data you made your choices on" — and it is far more recognisable than "tune". Keeping the familiar word. Also removes the labelling friction that prompted this. Step 5 previously told the skill to *offer* to add labels, which left the user with a mechanical JSONL edit and the balance to get right by eye. It now does the labelling itself and shows the resulting counts for objection, under two rules that are easy to get wrong manually: stratify within each polarity so both halves carry positives and distractors, and assign deterministically so the split never re-rolls between rounds — a reshuffled split is not a test split, because rows already tuned against leak into it. Verified end to end: `--split train` -> 17 rows, `--split test` -> 11, unsplit -> 28. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`!!! note` / `!!! danger` are mkdocs-material syntax. GitHub does not understand them: it renders the marker as literal text and turns the indented body into an accidental code block, so both callouts looked broken when read from the repo or a PR diff. Tutorials 01-07 use plain `>` blockquotes, which render correctly in both places — I had introduced the admonitions without checking the convention. Converted both callouts, and added a lint guard so the next tutorial cannot reintroduce them (mutation-tested: it fires on an injected `!!! note`). Scoped to tutorials deliberately. The one admonition under docs/ is in DATASETS.md, is site-first reference material, and predates this branch. Also drops `<sub>` tags from the skill's cost formula — a SKILL.md is read as plain text by a model, so HTML there is noise rather than markup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…table The execution track's instructions produced silently-wrong measurements. Every fix here is prose; no evaluation behaviour changes. - The outcome suite must be ONE dataset-backed task, one row per scenario. suite.json is written only for tasks the dataset expander touched (rollups group on suite_id), and --split filters dataset ROWS — so a directory of separate task files gives Stage A no rollup to rank and makes Stage C's `--split test` silently re-run the train rows. - Two consequences that decide how rows are written: criteria are copied to every row with ${row.*} substituted into every string leaf (so per-scenario assertions are parameterized, never written per scenario), and substitution never reaches sandbox.template_sources — every row shares ONE fixture, so variation lives in the prompt and the fixture must clear the skill's own hard stops or every arm ties at zero. - Step 4's worked example described a repo with no .github/, which is exactly the hard stop `ci` refuses on; it now shows a repo that clears the precondition and names its output path. - There is no --variant flag, so the arm set changes by authoring a file per stage: round<N>-triage/gate/confirm.yaml. Re-passing the triage file at Stage B/C costs (N+1)/2x the budgeted runs and renders no Paired Comparison block at all. - The paired mean diff subtracts in variant declaration order, so with incumbent declared first a candidate win reads NEGATIVE. Stated in both Stage B and Stage C. - Pin the tool policy under `defaults: agent:` and size the allowlist to the union of every arm's tools, or a candidate whose hypothesis is "use a different tool" is scored on the prohibition. - Name the per-row max_usd brake, and say plainly that an activation suite's tight caps must not be carried over — a truncated outcome row scores as a body failure that never happened. - Cost table: M_tune/M_holdout -> M_train/M_test, matching the split rename. The deletion sensor gains one entry per new instruction, each with its why, plus a counted assertion that the sign rule survives in BOTH stages and a guard that the stale cost-table symbols stay gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… track optimize-skill's execution track had no artifact to hand over: Step 4 pointed users at /coder-eval:task with a list of requirements attached, which came back half-applied. It now hands over a template, the way the activation track has always handed over activation.yaml. outcome.yaml is the execution twin of activation.yaml and carries the two constraints that are expensive to discover by running: - ONE fixture serves every row (${row.*} never reaches sandbox:), so scenario variation lives in the prompt and the fixture must clear whatever precondition the skill checks before it will act — else every arm ties at the floor and reads as three bad candidates. - Criteria are copied to every row, so per-scenario assertions are parameterized by row fields (path: "${row.expected_path}"). Its run_limits are deliberately unlike an activation suite's: an outcome row is a full task run, so max_turns is 20 rather than 2 and max_usd is the per-row brake. The run_command slot ships commented out — file_check already asserts what it would, and a row-supplied shell command would drag quoting and sandbox-tool-availability into every copy. Tests are loader-backed, mirroring the activation template's. The structural contract lives in ONE shared helper because Phase 4's checked-in sample asserts the same four properties; the threshold vocabulary is the union of a real aggregate() call and a real _attach_row_accounting() call, since completion_rate comes from the latter and a check against aggregate() alone would fail on the very template this ships. Also closes a harness gap found reviewing 1/8: nothing verified that a ${CLAUDE_PLUGIN_ROOT}/... pointer in a skill resolves to a file that exists, so the pointer at this template shipped one commit before the template did, past 344 green lint tests. Now 28 such references are checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…issed them docs/PLUGIN.md said "All six skills read it" while seven ship, and both READMEs named three of the four skills that shell out to the CLI. The count sensor was supposed to catch the first and did not: it substring-matched raw file text, and the offending line reads "All six\n skills read it", so the newline sat exactly between the two words the check needed adjacent. 91 lint tests passed over it. - docs/PLUGIN.md: six -> seven; both READMEs name all four members of SKILLS_REQUIRING_THE_CLI in the install paragraph; both `templates/` blurbs now name the outcome suite alongside the activation suite. - Extract `_normalized()` and read every prose surface through it. The idiom was already hand-copied at three sites, all now converted. - Extract the count matcher into `_wrong_skill_count_offenders` so its wrapped-phrase self-test can run the REAL matcher against a built file. Asserting only that `_normalized()` collapses whitespace left the sensor free to be reverted with every test still green — mutation-checked: reverting it now fails the self-test, and previously did not. - New `test_cli_driving_skills_are_named_in_the_install_prose`, derived from SKILLS_REQUIRING_THE_CLI with no names in the assertion, so a fifth CLI-driving skill cannot ship with the prose still listing four. Scoped to the install paragraph, since every skill name appears somewhere in both files. Note the strengthened sensor immediately caught this commit's own first draft ("the four skills that shell out to the CLI"). Reworded rather than exempted, as the sensor is deliberately blunt about any count word next to "skills". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…orked example Stands to the content track as tasks/skills/lint-tasks-activation.yaml stands to the trigger track: a real, commented, runnable suite built from the bundled outcome template. `ci` is the subject because it sets disable-model-invocation: true (so the rows MUST use the slash form, dogfooding the mechanism), its output is a real workflow file, and it needs no extra credentials in the sandbox. 10 rows, 6 train / 4 test. Because row substitution never reaches `sandbox:`, all ten run against ONE fixture, so each row is a different REQUEST against the same repository and each names its own output path — which takes the agent's filename choice out of the measurement without hinting at the content graded. The fixture is not scenery. It carries `.github/workflows/lint.yml` because `ci` stops outright on a repo with no `.github/` (an empty fixture ties every arm at zero and reads as three bad candidates), and that workflow avoids the string `coder_eval` or `ci` takes its don't-clobber branch instead. Its eval tree sits at `evals/` rather than `tasks/`, at two depths, with one task interpolating $SKILL_SOURCE_PATH, one experiment and a version pin — each making one load-bearing body rule observable rather than unreachable. The fixture lives under templates/, NOT tasks/ as drafted: tasks/**/*.yaml is swept recursively by CE034, test_yaml_migration, test_tags and the CLI's own default task discovery, all of which load every match as a TaskDefinition — and the fixture must contain an experiment file. templates/ is the repo's existing home for sandbox template dirs and is swept by nothing. One row was smoke-run for real ($0.43, SUCCESS, skill_triggered observed 'yes', file_check resolving a file the agent actually wrote) and it paid for itself twice: - max_usd was drafted at 0.50, which the measured row came within 15% of. A slightly longer row would have aborted COST_BUDGET_EXCEEDED and scored as a body failure that never happened. Raised to 2.00. - The emitted workflow invented `anthropics/coder-eval-action@v1` and still scored 1.0. The real action reference is now asserted as its OWN criterion rather than a second `includes` entry: file_check scores found/total, so folding a constant sub-check into the gated criterion would put a fixed contribution in every row of every arm and quietly weaken `mean: 0.7` on the one dimension that discriminates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

optimize-skill calls a partly-labelled dataset "the dangerous state, because it does not look like one", and it is right: --split keeps the rows whose label matches and SILENTLY DROPS the unlabelled ones, so the run succeeds, the report renders, and every metric is computed over a smaller suite than the file suggests. Nothing in the output says how many rows went missing. That is mechanically detectable, so per CLAUDE.md it becomes a rule rather than a paragraph. - Extract `row_split_label()` in task_loader.py as the single definition of the SPLIT-FILTER convention, and rewrite expand_dataset's inline predicate to use it in one pass. Behaviour-preserving: the old `x not in (None, "")` and the new `x is None or x == ""` were checked over None/""/0/0.0/False/True/[]/{}/ b""/"0"/" "/"None"/missing-key and agree on every one, so a falsy 0 remains a real label compared via str(). - _stratified_sample keeps its DIFFERENT convention on purpose (it folds a missing key into the "" stratum, which turns an explicit None into "None") and now carries a comment naming the divergence — the old comment read as though the two agreed. - CE035 as a @pytest.mark.lint class, mirroring CE034: every tasks/**/*.yaml with a dataset: block must label every row or none. Both legal states pass; only the mixture is a finding. It keys on the CONFIGURED split_field, not the literal "split", and covers inline rows as well as paths. Four dataset-backed repo tasks are exercised for real, not skipped. Also closes a second definition of "labelled" that had survived in the harness: the two template row tests used truthiness (`r.get("split")`), which would have reported a legitimate `"split": 0` as unlabelled — the exact disagreement this phase exists to remove. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rects the slash-form claim it disproved Ran the execution track against `ci` for real. Four baselines on --split train, ~24 runs, ~$9. Stages A/B/C were NOT spent: the round never cleared its own wiring check, and the method says a baseline that cannot be trusted is a stop, not a starting point. Full ledger in .optimize-skill/ci/history.json. Engagement never reached 1.0: 4/6, 4/5, 3/6, 4/6 across four runs, failing on DIFFERENT rows each time. Three silent slips, all observed: - the model answers the slash command by dispatching a sub-agent, which reads the skill in the child so no Skill call reaches the parent stream; - it ignores the command and does the work itself, emitting no Skill call; - the scenario's wording routes it to a sibling — "so we find out if a skill quietly stops triggering" sent one row to check-skill instead of ci. At that rate 20-50% of every arm would measure the absence of the thing under test, and Stage B's own promotion rule ("the skill actually engaged on every scored row") could not have been satisfied. Cost avoided: ~84 runs, ~$36. On rows where `ci` DID engage the weighted score is 1.000 — a ceiling. The emitted workflows carry the per-depth globs with their rationale, the real UiPath/coder_eval action, the version pin, extra-args, both runtime prerequisite steps and both hardening lines. So both no-go conditions hold at once. The round's real output is a correction to shipped guidance. SKILL.md and the bundled template both asserted the slash form "loads it, emits a real Skill tool call, and is detected by skill_triggered" — presented as reliable. Measured at 50-80%. Both surfaces now say so, name the three failure modes, and make engagement a GATE on the baseline rather than an afterthought. They also record that skill_triggered counts READING the skill's SKILL.md as engagement, not only a Skill call — one row reported engaged while the command it issued named a different skill. Suite and template fixes the run paid for: - `Skill` was missing from allowed_tools though it is the mechanism under test; it worked anyway, which is why the omission was invisible. - disallowed_tools: [Agent, Task] — an allowlist cannot suppress delegation, since those stay available whatever allowed_tools says. This one setting moved engaged-row scores from 0.333 to 1.000: with delegation on, the sub-agent did the work WITHOUT the skill and every row scored engagement only. The tool policy was not a detail, it decided the result. - turn_timeout 300 -> 900: 300 turned a row into an ERROR, which is excluded from the aggregate and surfaces only as completion_rate < 1.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… a null Written from the run in 5/8, which produced no promotion. Tutorial 08 already tells a "measure first, then decline to spend" story on the activation track; this is its execution-track counterpart, and the reason it stopped is more useful than a promotion would have been. The page covers the outcome suite's shape (one dataset-backed task, rows as scenarios), the one-fixture rule and why the fixture must clear the skill's own preconditions, criteria copied per row with row-parameterized assertions, split labelling, reachability, and the three checks to run before reading any score. Then the go/no-go, which was no-go on two grounds at once: engagement never reached 1.0 (4/6, 4/5, 3/6, 4/6 across four runs, failing on different rows each time), and where `ci` did engage the score was already 1.000. The sharpest exhibit is the same suite either side of one setting. With sub-agent delegation available, every row scored 0.333 and the emitted workflows named an action that does not exist, used the recursive glob the body forbids, and misspelled the score-floor input. Denying delegation moved engaged rows to 1.000. The tool policy was not a detail; it decided the result. It also carries the parts of the mechanics the round never reached but a reader still needs: the snapshot layout as a full plugin root (from the real incumbent snapshot, including why plugin.json is a trap), the per-stage experiment files, and the paired mean diff's sign — it subtracts in variant declaration order, so with incumbent declared first a candidate win reads negative. Every figure traces to a preserved artifact; the cost line is the measured $7.47, not an estimate. Two lint rules caught the draft and both were answered by rewriting rather than exempting: the recursive-glob guard (which is blunt on purpose, since integrators copy-paste out of docs) and CE026's runtime-prereq check on the emitted-workflow excerpt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…le it The page reported ~350 runs of A/B results without ever showing how the arms were wired: no snapshot layout, no experiment_id, no variants block, no `-e` invocation. A reader could follow the reasoning and not reproduce any of it. Adds, for the `analyze` round it already describes: - the .optimize-skill/analyze/1-<slug>/ snapshot tree, with the siblings and .claude-plugin/plugin.json, and why each is load-bearing — without the manifest the arms compete under different namespaces, and since a variant's plugins block REPLACES the task's, a snapshot missing its siblings makes every sibling criterion observe `no` in every arm; - the experiment YAML that mounts each snapshot by absolute path; - the exact command lines for Stages A, B and C, plus the rule that no flag selects a subset of an experiment's variants, so each stage needs its own file. Restructures into Part 1 (`lint-tasks`, a ceiling result and when to stop) and Part 2 (`analyze`, a full A/B that promotes), with the shared machinery under Reference. `lint-tasks`'s test-split confirmation previously sat between `analyze`'s Stage B and Stage C, so the page alternated subjects mid-argument; it now closes Part 1. Every finding survives verbatim — the recall-0.0 wiring trap, the 2-in-3 misfire, the billing-limit p = 0.038 artifact, and the uninformative test split. Now that the commands are shown, collapsing Stage B's three invocations into `--repeats 3` is a one-line edit, so a sensor pins it: three distinct --run-dir values, no --repeats in the fenced commands, and the warning itself retained. Scoped to the code fences, since the section legitimately says "not --repeats 3" in prose. Also corrects "the other six skills" to seven — tutorials are not in SKILL_DOC_SURFACES, so no count sensor covered that line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Cross-phase review (two external models plus an Opus reader over the whole diff) found three defects living in the seams between phases, where no single-phase review could see them. HIGH — the tool-policy instruction was shadowed by the artifacts shipped alongside it. SKILL.md said to declare allowed_tools/permission_mode in the experiment's `defaults: agent:`, but those fields merge by REPLACE and the task layer outranks experiment defaults — so the template and the sample, both of which declare them on the task, silently override it. Following the instruction was a no-op, and it broke precisely the case the paragraph exists to enable: widening the allowlist so a candidate whose hypothesis is "use a different tool" is scored on the instruction rather than the prohibition. All three surfaces now say to declare it on the suite, and note that a variant can still override deliberately. HIGH — four of the ten scenarios in the checked-in sample stated the behaviour they graded ("listed explicitly rather than with a recursive wildcard" scoring the explicit glob; naming the skill-source requirement and scoring its passthrough). An arm whose body DELETED that rule would still have scored 1.0, because the prompt supplied it — the answer-leak this repo's own task rubric and lint-tasks flag. Rewritten to describe the situation and leave the method to the body. Tutorial 09 now carries the caveat that its reported figures predate the fix, which makes its ceiling reading softer rather than firmer. MEDIUM — the template still shipped max_usd: 0.50, the cap a measured $0.43 row came within 15% of, while the sample had been raised to 2.00 from that same evidence; a comment claiming allowed_tools is a claude-code field when it is on BaseAgentConfig (only setting_sources needs the discriminator); SKILL.md listing "keep Skill in allowed_tools" as an engagement mitigation when tutorial 09 measured that it works either way; and SKILL.md conflating the two cap failures — max_turns exhaustion scores low (a fabricated body failure) while a timeout ERRORS and is excluded, surfacing only as completion_rate < 1.0. Test harness: the fixture depth assertion counted evals/experiments/ as a depth, so deleting the file it was entirely about still passed (mutation-verified fixed); _normalized's docstring asserted a repo-wide invariant eight sensors do not follow; _outcome_metric_vocabulary was documented as universal while derived from file_check alone, which would have failed the next gate the docs ask for. Also records why the fixture lives under templates/ rather than beside its suite, and names reference/templates/ in CLAUDE.md's plugin inventory. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… grade Closes the harness gap behind one of the final review's High findings. Four scenarios in the checked-in outcome sample stated the behaviour they graded, so an A/B arm whose body DELETED that rule would still have scored 1.0. This repo's task rubric and the `lint-tasks` skill both call that out — for a USER's files. Nothing applied it to this repository's own, so it shipped. Guards the verbatim form: a criterion's content-asserting strings must not appear in the prompt the row expands to. Location fields (path, agent_file, command) are exempt, and that distinction is the rule — a prompt MAY say where to write, which takes the agent's filename choice out of the measurement without revealing anything graded; it may not say what the artifact must contain. Scope is stated honestly in the docstring: this cannot catch a SEMANTIC leak, where the prompt describes the graded behaviour in different words. That is what actually happened here and it needs a reader. Guarding the blunt form is still worth it — it is the easy mistake and it is silent. Mutation-verified: injecting `minimum-task-score` into a scenario that grades it fails the rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… named function `_activation_preflight` holds both halves verbatim: the cross-split refusal that compares what the two run.jsons SAY, and the tree reconciliation that asks whether either describes the tree it sits on. This is a move, not a rewrite — every pre-existing preflight test passes UNMODIFIED, which is the extraction's own acceptance criterion. It returns a refusal STRING plus notes rather than a verdict, because building one needs `scored_row_ids`, `rows_excluded` and the caller's `notes` list, none of which the preflight has and all of which `_refuse_activation` closes over. The caller EXTENDS its notes rather than re-binding them, and that is load-bearing: `notes` is the same list object `_load_and_pair` returned, and pydantic copies it at construction — so `notes = notes + preflight_notes` would leave every later append landing in a list no verdict ever sees. A test now pins that a note written before the preflight and one written after it both reach the same verdict. Refusal precedence is program order, and it is now tested rather than implied: a pair that is both cross-split AND contaminated reports the cross-split cause, the more specific one, whose remedy is actionable without first understanding the other. `activation_gate` goes D(29) -> D(23). It stays D deliberately: what remains is the gate's own work — the sibling scan, the guardrails, the MDE, the all-negative-subset note, the bootstrap and two verdict constructions — and splitting further would separate the statistic from the block that reports it. Nothing in the module is E-grade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…se 7 needs it The skill's 11 `coder_eval` import lines are what a USER runs. A broken one fails in their terminal *after* they have paid for the runs the snippet reads, and the existing guard was blind to it for two independent reasons: a moved name resolves to `None`, which the call loop skips via `if not callable(target): continue`; and that loop only ever visits names used as `ast.Call` funcs, so a name imported but never called — `CostQualityPoint`, `SearchComparison`, `TASK_JSON_GLOB`, `GATE_RESAMPLES`, `MATERIALITY_FLOOR`, all of them in the shipped skill — never reached it at all. So the existence check runs over the import map, not inside the call loop. Rehearsing Phase 7 against the real file then found a third hole, in the shipped code rather than in the plan: the call loop's own `import_module` was UNGUARDED, so a mistyped module name raised `ModuleNotFoundError` straight out of the sensor instead of reporting a failure — the precise thing this sensor exists to prevent, on the precise change it exists to protect. Names are now resolved once, in the existence pass, and the call loop reads that map. Re-pointing every import at a module that does not exist now reports 19 failures instead of crashing. Anti-vacuity throughout, because a sensor that cannot fire is indistinguishable from a clean tree: both holes get a test, both `origins` regexes (parenthesized and single-line) get one, a real non-callable import must stay silent, and the wholesale-rename rehearsal is pinned against the real `SKILL.md`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… track The boundary is DERIVED from the module's own call graph, not chosen: every one of the 79 top-level names is assigned, and a solver over the whole graph reports ZERO illegal edges at the ranks `load(0) < gate(1) < {activation, execution}(2) < {fronts, search}(3)`. A test now asserts those RANKS rather than mere acyclicity, because `activation -> execution` would be perfectly acyclic and still wrong: the two tracks are meant to be independently readable, which is why the split is BY TRACK. optimize_load.py 713 loading, pairing, provenance, row primitives optimize_gate.py 374 the decision primitives BOTH tracks share optimize_activation.py 1159 the activation track, end to end optimize_execution.py 1033 the execution track, end to end optimize_fronts.py 327 the three Pareto fronts optimize_search.py 286 the search loop + the leak preflight No E-grade function anywhere in the family, and the seven D-grades spread across four modules with a maximum of three in any one. Moved verbatim: no behavioural test changed. The five failures the move produced were all layering or module-reference assertions, which is the plan's own signal that nothing was rewritten on the way. `test_a_moved_name_is_gone_from_the_gate` became `test_a_moved_name_lives_in_exactly_one_module`, widened from two modules to the whole family — it was already derived on both sides, so it covered five new modules for free. That test is what forbids a re-export facade, and a facade is what would have made this cosmetic. `optimize_activation.py` overshoots the plan's 1,100-line criterion by 59 lines. The estimate was measured at the pre-Phase-1 spike; Phases 1-5 then added `_activation_notes`, `_activation_preflight` and their rationale to that track. Reported rather than fixed by contorting the partition to hit a number that predates five phases of work. CLAUDE.md's entry for this module was 20,506 characters — a directory diagram that had stopped functioning as a map. It is now six short entries totalling 3,266, with the rationale where it belongs: the module docstrings and the models' field descriptions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`render_markdown` keyed its BLOCKED headline on `verdict.guardrails` alone while its execution twin already unioned `integrity_checks`. So a candidate that separated, cleared Holm, and was vetoed by a failing SIBLING check had `promoted = False` and rendered NOT PROMOTED — indistinguishable from one that simply lost. Reproduced on identical verdicts: activation NOT PROMOTED, execution BLOCKED BY A GUARDRAIL. That is the exact confusion this rung exists to prevent, and it survived a change whose whole purpose was to make the two tracks mean one thing by `promoted`. "It won and was vetoed" and "it lost" call for opposite next actions. Found independently by two reviewers — raised as a Low in the Phase 2 review and deferred on the grounds that fixing it moves a pinned fixture, then raised again as a High in the final cross-phase pass. The fixture cost was never an argument about correctness. The headline names a failing sibling check while the note ladder still does not add a generic `FAILED — this forces` line for one. That asymmetry stays: the sibling rung already writes a note saying the candidate "moved the failure rather than fixing it", which says more than the generic sentence would, and printing both would say it twice. One pinned line changes — the headline — on a verdict whose `promoted` was already False. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The prose-sensor family asserts tokens are PRESENT. Nothing asserted that a claim the code RETIRED is absent — and that gap cost this repo twice in one change. Folding the cost/latency guardrails into `promoted` meant deleting "the guardrails gate here, in the procedure" from six surfaces; a SEVENTH, the activation-track twin in SKILL.md, was missed by the plan's own enumeration AND by every existing sensor, and survived until a reviewer read the file. A stale sentence is bad. This kind is worse: it INSTRUCTS. A reader told the guardrails gate in the procedure goes and checks `.passed` by eye on a field that has already decided it. Verified against the text that actually shipped: run over `ae9aea2`'s SKILL.md the sensor fires on all three retired claims, including the one the enumeration missed. Anti-vacuity feeds each fragment through the real `_normalized` reader HARD-WRAPPED, because these documents are wrapped and a raw substring check passes on exactly the text it exists to catch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

**A refused execution block still printed "an ordinary negative result".** Phase 3 guarded the activation ladder and CLAUDE.md then claimed a refusal suppresses every negative-result note on BOTH. It did not: the below-MDE `else` branch — the one that calls itself "an ordinary negative result and not a measurement problem" — was the single note in `_execution_diagnostics` with neither guard, while the two advisories under it had both. Reproduced through the real gate: a zero-variance refusal printed it beneath NOT A RESULT. `promoted` was unaffected, so it was prose only — on the page a user pastes into a ledger. The existing suppression tests could not reach it because their contamination fixture also nulls the floor, so the new test drives the branch with `mde` set. **Three scans went vacuous in the split**, each reading `optimize_gate.py` alone after its subject moved out of it. `test_the_trim_is_declared_once` and `test_neither_wrapper_respells_a_shared_note` both passed under mutation — an inlined `min(len(a), len(b))` and a respelled shared note were invisible. They now read the whole family through one `_family_source()` derived from `_OPTIMIZE_RANKS`, so a seventh module joins every scan at once. **Fourteen files carried stale `optimize_gate.<name>` paths**, including two shipped surfaces an agent acts on: `reference/proposal-prompt.md` told the proposer to call `coder_eval.optimize_gate.candidate_leaks(...)`, which now raises, and two Pydantic field descriptions pointed at moved functions. Phase 6's sensor only resolves fenced snippets, so prose and docstrings went unchecked — recorded as a harness candidate. Also: `docs/REPORT_SCHEMA.md` attributed two watched constants to the wrong module (the parity test checks names, not attribution); the "3,521 lines" figure matched no commit (it was 3,291 at the plan's baseline and 3,664 immediately before the split); and two prose claims overstated what the code does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The six-module split left fourteen files naming a moved symbol by its old dotted path, including a shipped plugin reference an agent acts on and two Pydantic field descriptions. The snippet sensor resolves fenced imports only, so every one was found by a reviewer reading files. Recorded with the shape of the guard and why it is not a five-minute job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…be mounted `coder-eval plan` is the surface `/coder-eval:task` tells an author to validate against, and it printed ✓ on a suite whose `template_dir` fixture does not exist. The failure surfaced only at sandbox setup — `RuntimeError: Template directory not found` — after the run had started and tokens were being spent. Measured on the plugin's own `reference/templates/outcome.yaml`, which mounts `./outcome-fixture` and has no such directory. Both questions are asked through the sandbox's OWN predicates, extracted here rather than copied: `template_dir_problem` (the two raises in `_apply_template_dir_source`) and `escapes_sandbox` (the rule `_resolve_within_sandbox` enforces). A hand-copied second implementation drifts silently in both directions — a sandbox that gains a third precondition would leave `plan` blessing a suite that cannot run, which is the defect being closed. `starter_files` is checked for the failure it actually has: it carries inline content, so there is nothing to stat, but its DESTINATION can be absolute or `..`-escaping and no model validator catches that. `repo` has nothing local to resolve. Destinations are checked against a synthetic root, since no sandbox exists at plan time. An undefined `$VAR` in a template path is left exactly as it was — a load-time `ValueError` from `resolve_template_source_paths`, exit 1 — and pinned by a test asserting the new check is NOT what reports it. The plan asked for it to be a warning; that would mean loosening the run path, which is out of scope. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`/coder-eval:optimize-skill`'s execution track says "no suite -> stop and point at `/coder-eval:task`. Do not author one here" — the rule exists because a suite written by the thing that will judge it is fitted to it. The instruction was unfollowable: `task` had zero occurrences of `dataset`, `jsonl` or `row`, and its guidance was explicitly one-file-per-task, which is the precise anti-pattern (N task files produce no `suite.json` rollup, and `--split test` silently re-runs the train rows at full price). The mode is prose only — no criterion, no model field, no evaluation path. It supersedes the one-file-per-task guidance in both places it is READ, not only at the top of the file, and it points at `reference/templates/outcome.yaml` for the suite's fields rather than restating them. What it adds is the part the template cannot carry: Step 2.5's rule inventory, which is what makes the rows derived from the skill under test rather than invented beside it, and a deterministic split. The split pattern is written out (`train, train, train, test, test` down the id-sorted list) rather than described as a ratio: the first draft said "alternate, roughly 60/40", and alternating is 50/50. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The execution track had no in-repo prior art for a continuous grader: the checked-in worked example scores with `skill_triggered` + two `file_check`, and the bundled template's third criterion slot taught a BINARY `run_command` — which over a dozen-odd rows manufactures the execution gate's zero-variance refusal, two arms of genuinely different quality scoring identically. `verify.py` is that instrument, stdlib-only because the sandbox venv installs what the AGENT needs, not what the grader does. Its contract lives in its module docstring; three parts of it are load-bearing and were found by running it, not by reading it: - **An N/A trigger must be a property of the ROW, never of the artifact.** The first draft returned NOT APPLICABLE when the artifact was not JSON, which makes the denominator a function of the arm's own output: an arm that ignored the requirement scored 1/1 while an arm that complied and got one field wrong scored 1/2. The worse artifact wins, and that can invert an A/B verdict rather than merely bias it. - **Params are validated.** `"all_of": "one needle"` — the most natural slip in a hand-written expectations file — iterated CHARACTERS and reported "all present" against any artifact of moderate length. - **Exit 0 always**, because `_score_from_stdout` checks the exit code before it parses line 1: a non-zero exit throws away a score the grader already computed. The shipped `outcome.yaml` mounts a fixture that did not exist, so Phase 1 made it fail its own `plan`. It ships one now — a placeholder README — rather than an obviously-broken path: the template's first `plan` must be green, or the signal Phase 1 added is spent before anyone uses it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…a stage An unfair grader is the one error class an A/B cannot detect: it biases every arm EQUALLY, so the ranking, the paired test and the confirmation all agree with each other and are all wrong together. In the last dogfood round the hand-written grader was wrong twice and neither error was visible in any comparison. So `/coder-eval:task` gets a Step 6.5 that grades a known-good and a known-bad artifact by hand and reports the separation margin. It also catches, for free, a row that declares only checks that do not apply to it: the known-good artifact scores 0.0 there with `0/0 applicable`, instead of the row quietly dragging an arm's mean down later. The grader-fairness questions move to `reference/task-rubric.md`, which is where the plan put them — but they already existed verbatim in optimize-skill, which the plan did not know. Two copies of the checks that exist to catch what nothing else can is the wrong answer, so the rubric declares them once and both skills point at it, guarded in both directions: `RUBRIC_READERS` keeps the pointer, and a new test asserts neither skill restates the questions. The rubric also gains the outcome-suite spec carve-out — a prompt MAY name the output path and format, because "follow the user's spec" is itself graded, and MUST NEVER name a rule the body is supposed to supply — with the test that separates them: could an agent WITHOUT the skill satisfy the check from the prompt alone? Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Everything under a mounted `template_dir` is copied into every sandbox, so a grader's `expectations/` placed there hands the agent exactly what it is being marked against — and the run still looks completely normal. Measured when it happened by accident: on the same 11 rows the mean went 0.9158 clean -> 0.9461 leaked, which is larger than most effects an optimization round exists to detect and inflates every arm, so no cross-arm comparison reveals it. Asserted as a test rather than a numbered lint rule, on purpose. A tree-walking rule has exactly one discoverable subject today, and that subject's fixture is a placeholder directory — so it would report clean whether or not it worked, which is the CE044/CE045 vacuous-pass failure. CE056 is reserved instead, with the promotion trigger written down: a second outcome suite carrying a `run_command` grader. The test classifies every member of the `TemplateSource` union and FAILS on an unrecognised one, asserts the discovered mount set is non-empty, and asserts the path relationship whether or not the mount exists on disk — three ways this assertion could have become the vacuous pass it exists to prevent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two independent reviewers (gemini-3 + gpt-5.3-codex in parallel, and an Opus sub-agent) over the whole five-phase diff. Nothing Critical or High survived verification; these are the Mediums, plus the Lows that produce a WRONG SCORE rather than a cosmetic defect — which in a measuring instrument is the same severity by another name. **`plan` validates the variant layer too.** `template_sources` merges by APPEND, so an experiment's `defaults:` or a variant can add a mount the task never declared. Checking the task's own sources left `plan -e <exp>` printing "All tasks are valid!" for a suite whose run dies at sandbox setup — the exact failure Phase 1 exists to move earlier, one config layer up. A `reported` set keeps one bad path to one line rather than one per variant, and the per-variant loop moved to `_report_variants` (the statement ceiling was the forcing function; it is also where per-variant checks belong). **`check_json_field` no longer penalises the richer artifact.** Searching every depth and requiring all occurrences to match failed `{"status":"ok", "meta":{"cache":{"status":"cold"}}}` — an arm marked down for emitting MORE detail, which is failure mode one of the rubric this change shipped. It now answers from the shallowest occurrences only: relocation still passes, repetition at one level still has to agree, and unrelated deeper keys are not answers. **The grader is addressed through `$TASK_DIR`, not an absolute path.** The template taught a hardcoded host path, which does not exist on a colleague's machine, in CI, or under `driver: docker` (which mounts the task dir but not an arbitrary path). All three failures score 0.0 on every row of every arm — indistinguishable from a bad skill body. Also: one check is one detail line (a newline in a detail could forge PASS/FAIL lines); `utf-8-sig`, so a BOM'd JSON artifact is not scored as prose; `field` validated like `all_of` was; the unreachable `checks` guard reordered; and the harness backlog reconciled — the instrument-fairness entry is ticked (Phase 4 shipped it), and the answer-key entry now cross-references CE056 instead of standing as a second copy of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…e it is paid for The shipped template failed the rules it teaches: four rows against one expectations file matching none of them. Author one file per row, each at the four-check floor, and delete the example whose content they inherit. Then the two authoring-time rules a validating file cannot express — a row's role (a guard at 1.000 can only show regression; rows with headroom discriminated 71% of the time against 12% for rows at ceiling) and the bidirectional rows/expectations parity whose missing half scores a hard 0.0000 that reads as a catastrophically bad arm. Step 7 projects run counts from optimize-method.md's cost table rather than restating it: CE039 does not cover task/SKILL.md, so a restated number would ship with no sensor at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The ceiling arithmetic in Phase 3 needs to know which rows fail which rule. Nothing carried that: the grader printed `FAIL R1 …` as prose. So an optional top-level `rules` map — a sibling of `checks`, never a key inside one, because params are forwarded verbatim to the check function where an unknown key is a silent extra argument — and one machine-readable line at the end of stdout. Any-fail, all-na. The direction is the load-bearing part: any-fail counts the most rows as failing a rule, so a headroom estimate built on it is an UPPER bound, and "this rule cannot clear the floor" holds even under the most generous attribution. That is the only claim the table will make. `_report` owns the line, so the ten early exits carry it too — a consumer must be able to tell a crashed grader from one that predates the contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…one draw is worth THE CEILING. A candidate for rule R can only gain where the incumbent lost, so the rows failing R bound the whole effect — and the suite mean divides that by EVERY row, including the ones already at ceiling. Measured on a real 15-row round against a 0.0255 floor: R1 0.0300, R6 0.0223, R7 0.0191, R8 0.0095. Three of those four candidates were written, run and gated anyway, at roughly $40, off inputs already paid for. The arithmetic was available beforehand. `rule_row_map` inverts the grader's RULES line into rule -> failing rows, and `headroom_ceiling` is ONE function for both the suite-level and the rule-level question because they are the same arithmetic over different subsets. The denominator is the full row count: dividing by the rows that failed reports a per-row lift and makes every rule look promotable. It is advisory and gates nothing — the attribution is authored, and a mistyped rule id must not be able to veto a real promotion. RENDERING HONESTY. A single-replicate matrix ranks; it does not measure. One reported +0.0392 against a 0.0255 floor and pushed the incumbent off the Pareto front, while the replicated gate over the same rows returned p = 0.9977. And a suite mean cannot say that one row went 0.76 -> 1.00 while another went 0.86 -> 0.59, both perfectly reproducibly, cancelling to +0.0001 — which is the opposite of "the difference is noise". CE039 now recomputes the Step 7 table from the real vector, so the table, the estimator and the round cannot drift apart in pairs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A mid-round grader fix moved a suite mean 0.8679 -> 0.9158 on IDENTICAL artifacts, and nothing in any run directory recorded that the instrument had moved. The improvement was indistinguishable from a body that got better. `verify.py --fingerprint` hashes the script plus the expectations it LOADS — the answer key is part of the instrument, a `.DS_Store` or a `__pycache__` is not, and a filter naming today's stray files is a list nobody can finish. It also exits NON-ZERO on failure, alone among this script's paths: everywhere else exits 0 to protect a score already computed, and here a score-shaped line is what the caller records AS the fingerprint. `RoundScores.grader_fingerprint` is optional, so an existing measurements.json still parses — a MISSING key is the field default, which is a different mechanism from `extra="forbid"`. `grader_changed` is three-valued and `None` means UNKNOWN, never False: a round that recorded no fingerprint must not be able to masquerade as an instrument that provably did not move. Reported, never enforced. A changed instrument makes two rounds incomparable, which is a fact about the measurement rather than a veto on a promotion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…pectations grade CE036 asks this of a criterion's own fields, which makes it blind to exactly the suite that most needs it: on an outcome suite the criterion is a `run_command` naming a script, and every graded string lives one indirection away in `outcome-grader/expectations/<row id>.json`. So the marking scheme a paid optimization round reads was the one thing nothing checked. A leak there is worse than an ordinary one — the prompt supplies its own answer, so the row scores well whether or not the behaviour happened, and in an A/B an arm that DELETED that behaviour still passes. It biases every arm equally, so nothing downstream can reveal it. The non-vacuity assert counts prompt/value PAIRS, not discovered suites. Before this plan the only in-repo suite had one expectations file matching zero rows, so a suite-level assert would have passed while comparing nothing — which is the CE044/CE045 vacuous pass this rule cites by name. It compares 16 pairs on the shipped template today, and catches a planted leak in it. Third consumer of leak_detection.py, not a third implementation. Nothing under tests/lint/rules/, runner.py untouched, CE056 still reserved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… checking CE039's new headroom-ceiling claim passed over a table trimmed to one row — every remaining cell recomputed correctly while the table said the opposite of what the claim exists to assert. Fixed there by hand; the general form is a mutation test over the registry, and it is deferred because a fair perturbation has to be claim-shaped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ut what it does not know Five findings, all in the layer that only ever REPORTS, which is exactly where a wrong number is hardest to notice. The stderr window was defeatable by the input it names. Ending it at the LAST `Stderr:` marker meant a traceback quoting artifact text could contain a second one and move the boundary back past the grader's real attribution. Both markers are now the FIRST, and the untrusted side is outside the window either way. CE057 fired on the shape the shipped template teaches. `expected_snippet` is a criterion parameter, not a prompt: the template tells authors to put the graded string there, so a suite doing exactly that reported a leak with the string in the prompt zero times. It now renders the real `initial_prompt` — which also catches a leak hardcoded into the shared template, invisible before. A row whose grader output lost its RULES line is in no rule's failing set, so every ceiling is an UNDER-estimate — and `GAP`, the verdict that says stop working on this rule, could be produced by a stdout truncated at 4000 chars. `rule_row_map` returns those rows now, and the block says so. It also keys every rule SEEN, so the rule a suite always passes can be sized (ceiling 0.0) instead of vanishing from the table. And two the tests knew about: the replicate evidence printed deltas that did not follow from the triples beside them, and the caveat sensor checked two things separately rather than binding the constant to the prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every High in this plan's reviews was a guard that did not guard, and the one that generalises is the scaffold's: `verify.py` is 460 lines of executable Python that ships to users, exercised only through subprocess, and outside `source = ["src/coder_eval"]`. Nothing measured whether its branches were reachable. Measured before writing this — an added `if "this-can-never-match" in path.parts: continue` passed all 1331 tests. So drive it IN-PROCESS and assert no line is unreachable. It reproduces the original defect: re-introducing the dead `__pycache__` filter fails with the offending line quoted. Writing it found 13 paths nothing in the suite had ever executed — the absent JSON field, the list branch of the breadth-first walk, a malformed `rules` block, a non-string rule id, the unreadable-artifact OSError, and the dispatch's handling of an author's check that prints or returns a non-bool. That last pair is unreachable with the two sample checks, which is a fact about the sample vocabulary rather than the dispatch, so the drive registers checks — exactly what the file tells an author to do. The other two Highs are deferred with the reason each is not cheap to guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…o keep it stamped `weighted_score` is the execution gate's primary statistic and a weighted mean over every criterion, but nothing in a run artifact recorded the blend — so the number the gate decides on was not reconstructible from the file it is read beside. `CriterionResult.weight: float | None = None`, `ge=0.0` and `allow_inf_nan=False` (both bounds, matching the criterion side: `ge` alone lets `inf` through, and an `inf` weight serializes back as `null`, collapsing the very distinction the field makes). `None` means NOT RECORDED and is deliberately not a recorded `0.0` — the `RunSummary.row_selection` precedent, not `gating`'s benignly-lying default. Stamped at all three `checker.py` sites. Nothing under `criteria/` changes: every `CriterionResult(...)` there is downstream of the seam. CE058 makes that permanent. `CriterionResult` now mirrors three fields, which is a recurring class rather than a one-off: a fourth added to the model and not stamped is silent — every result carries the default, nothing raises, no type checker complains. The rule requires a `"mirrors "`-described field to be assigned in EACH of the three functions, not any one of them; a union is satisfied by `_finalize_result` alone, which is exactly the shape that leaves the two error paths defaulting. It is class-wired rather than a `BaseRule` for a structural reason — `BaseRule` visits one file and this predicate spans two — so it follows CE057 exactly: body in `tests/lint/mirrored_result_fields.py`, `tests/lint/rules/` and `runner.py` untouched. Review findings fixed in place: the any-of hole above; a false "ONE stamping seam" docstring line that directed a future author into the uncovered shape; and a subclass-scoping test that was empty by construction — it now asserts on a subclass field that DOES carry the marker, the only input separating "reads the base model" from "walks every subclass". Two test doubles in `test_evaluator.py` gained `weight`: they stand in for a `SuccessCriterion`, and a bare `Mock` is not a valid float. `docs/REPORT_SCHEMA.md` and `CLAUDE.md` also carry Phase 1's entries in this commit — one file each, and the two phases land back to back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… is auditable `weighted_score` is a weighted MEAN, so a criterion that scores identically on both arms on every row contributes its whole weight to that mean's denominator and nothing to its difference. The shipped outcome template does that by design — engagement and the `file_check` both saturate — so an effect confined to the grader reaches the gate multiplied by 1/2.05, and nothing anywhere said so. `ExecutionGateVerdict.dead_weight` is that share, with the naming sentence in the existing `notes` list rather than a second field. Computed beside the noise floor, before `_verdict` exists, so every return path reports it and the note is in the list pydantic copies at construction. **It is a READING and can never gate**, and that is a permanent decision rather than a deferral, measured rather than argued: a constant criterion scales the paired difference vector without changing its shape, so it scales the mean AND the standard deviation by the same factor — the paired *t* is identical to 1e-12 between the grader-only and blended scales while the mean difference scales by 1/2.05. Every conjunct of `promoted` is invariant to it, so wiring it into `integrity_checks` would force `promoted = False` on comparisons that are statistically sound. The one case where dead weight genuinely invalidates a comparison — every criterion constant — is already the zero-variance refusal. `None`, never `0.0`, on five distinct causes, each naming itself in `notes`: fewer than two rows paired, an arm with no criterion results, arms whose criteria lists disagree (a contaminated tree, whose diagnosis the reconciliation refusal owns), an unrecorded weight, and zero total weight. The rendered UNKNOWN names no cause, because only the note knows which of the five it was. Replicates collapse by MEAN before pairing, matching `paired_comparison` and `arm_row_scores` so the three surfaces agree about what a row scored, and the `== 0.0` test carries no tolerance — a tolerance would reclassify a small real effect as unmeasurable attenuation. Both are pinned, the second by a 1e-9 difference that must read ALIVE (verified by mutation: `abs(diff) < 1e-3` fails it). `_criterion_weights` is private beside `_row_score`, which is REUSED for the per-criterion extraction rather than re-derived. Pins regenerated: one key, one note and one rendered line each; no statistic moved. One ADDED `## Estimator changes` row (in the previous commit, with the schema ripple). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…st instrument provenance The grader fingerprint covers the outcome track's script and its answer key. It cannot see a criterion `weight` change that re-blends `weighted_score` — the number the execution gate's paired *t* compares — and on the activation track it answers `None` every time, because that track has no script grader at all. So the suite around the grader had no provenance, and on activation nothing did. `suite_fingerprint(task, rows)` is a SHA-256 over the criteria (each concrete subclass dump minus a reason-carrying denylist of ONE, `description`), the prompt template as authored, the expanded rows the round scored, and the whole `run_limits` block. `RoundScores` records it; `optimize_store.suite_changed` compares it, three-valued and sharing one body with `grader_changed` — `None` means NOT RECORDED, so an older sidecar cannot masquerade as a suite that provably did not move. **Its own module because the store PERSISTS and this COMPUTES**, the precedent `leak_detection.py` and `reports_optimize.py` set. Not a cycle — `optimize_store` imports `coder_eval.models`, `TaskDefinition` is in it, so a digest living there would close nothing. The review of the first implementation changed what this covers, and the corrections matter more than the addition: - **The rows are hashed, not just their ids.** `activation.yaml` is `initial_prompt: ${row.prompt}` with `expected_skill: "${row.expected_skill}"`, so every prompt AND every label lives in the rows file. A digest over ids alone left a rewritten prompt and a flipped label byte-identical — the commonest suite edit there is, on the one track this digest is the sole provenance for. The grader half hashes `expectations/*.json` because the answer key is part of the instrument; this is that rule on the other track. - **`run_limits` is hashed whole.** Curating four caps was justified by a claim that is false twice: the three token caps abort a run exactly as `max_usd` does, and `stop_early` is the kill switch for every armed criterion, so it moves `f1.yes` itself. - **The length prefix and the section tags are redundancy here, and the module says so.** Verified by mutation: removing either leaves every test green, because canonical JSON per part already escapes the delimiter and quotes strings. They stay for locality; the false attribution does not. - **The dump-settings tests asserted digest inequalities and caught nothing.** Each of `exclude_none` / `exclude_defaults` / `exclude_unset` left all 41 tests green — dropping a key from one side of an inequality leaves the sides unequal, and the real damage is COLLISION (`exclude_defaults` drops the `type` discriminator, after which two criterion types hash the same). They now assert the contract on `scoring_dump`, and all three mutations fail. - Step 11's snippet referenced a variable defined only inside a comment — a `NameError` in the user's terminal after the round is paid for. It is live now, uses the UNION of arms' row ids rather than `arms[0]` (a hole is absent, so one arm's vector is shorter), and was executed end to end against the shipped suite. - Step 11 also stops printing "comparability unknown" on every activation round: that track has no script grader, which is not the same as not knowing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`GATE_RESAMPLES` is DERIVED from `GATE_P_PRECISION` at the strictest Holm threshold for `GATE_MAX_FAMILY = 5` survivors. Above that S the threshold tightens while the draw count does not, so the Monte-Carlo error of the p stops being the declared fraction of the threshold it is decided against — and nothing said so: the block printed the family size while the declared precision sat on a constant nobody reads at that moment. `_note_resolution_degraded` states the strictest threshold, the precision actually achieved, and the draw count that would restore it. It returns `str | None`, unlike `_note_holm_family` beside it: putting the threshold condition IN the function keeps it in one place, where an `if` duplicated at the two call sites is the shape that lets the two tracks drift — which is the whole reason these notes are shared. Emitted at both existing call sites, OUTSIDE the negative-result guard beside the family note: it is a statement about the draw count, not a claim about the candidate, so it stays true under a refusal. Both wrappers report the family's SMALLEST `n_resamples`, read off the verdicts rather than from the constant — the coarsest member is what bounds the family's resolution, and only members with a p were tested at any resolution. No constant moved and no decision changed: `gate_refusal`, `promoted` and `separated` are untouched, asserted directly. **No existing pinned fixture changed and no ledger row is owed.** Every pinned render carries a family of 1 or 2, so the note appears in none of them; it is pinned by a NEW family-of-8 fixture, which has no "before" and therefore no step to attribute. The test says so, so that if it ever starts modifying a sibling the reach is visible. Every figure in the tests is recomputed from the shipped constants. The table they reproduce was verified against the working tree first: S=6 → 0.1095 / 24,000; S=8 → 0.1265 / 32,000; S=10 → 0.1414 / 40,000. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A candidate is free to edit `scripts/` and reference files, and `reference/proposal-prompt.md` is about to say so out loud. A graded string bundled into one of those was invisible to `(arm / "SKILL.md").read_text()` — and the preflight then returned **clean**, byte-identical to a genuinely clean candidate, which is the worst shape a check of this kind can have. `skill_text(skill_dir)` concatenates every UTF-8-decodable file under the directory in sorted relative-path order, each preceded by its path — so a finding stays locatable and two files cannot concatenate into a phantom match across the boundary between them. `candidate_leaks` stays PURE: two strings and a row list. Widening the scan inside it would have made it read the filesystem, so the IO is a separate function and the module docstring says which half does what. Its boundary list is FIVE now; the fifth is that the caller decides what text is scanned and the checker cannot tell how much of the candidate it saw. Skipped, and each verified by execution rather than assumed: a non-UTF-8 file (a binary cannot carry a verbatim graded string in the form `graded_strings` produces), an `OSError` (a permission, or a file that vanished mid-walk — a preflight must not abort a round over one stray file), a symlinked FILE, and a symlinked DIRECTORY. The last two are stopped by DIFFERENT mechanisms — the explicit `is_symlink` check, which must precede `is_file` because that one follows the link, and `rglob` not recursing through a symlinked directory — so each has its own test. The flagged-in-scripts test asserts BOTH halves: the tree read flags it and the one-file read does not, so the test states the defect rather than only the fix. And a span already in the baseline's own reference file stays unflagged, so widening the scan cannot re-introduce the wolf-crying the diff was built to prevent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…er candidate space Stage C was prose: "report that block verbatim alongside the test F1s", leaving the reader to eyeball two intervals and decide whether one reproduced the other. It now has a verdict. `confirm_gate` (activation) / `confirm_gate_execution` (execution) run the track's own gate on the confirm run directory and classify the train→test delta as REPRODUCED, SHRANK, REVERSED or UNDECIDED. `ConfirmVerdict` carries both effects, the delta, the confirm split's own MDE and the full confirm-gate block; `render_confirm_markdown` prints it, with REVERSED directly below a refusal because a reversal is a headline rather than a footnote. Four things it deliberately does NOT do. It measures no floor — `execution_gate` already prices the replicate null split on whatever run dir it is handed, so a second estimator would be the CE037/CE040 defect class and would double every confirm's cost. It confirms exactly ONE candidate, raising on a list, because confirming a shortlist spends the held-out split on SELECTION. It applies Holm at m = 1 only so the carried block reads as a decision — there is no multiplicity at Stage C. And it REFUSES a confirm run that did not record `--split test`: a `train` confirm re-runs the rows the candidate was fitted to, reproduces by construction, and costs full price with no error anywhere. `classify_confirm` and `build_confirm_verdict` live at rank 1 with three more shared primitives, because the two rank-2 track modules may not import each other and per-track means two copies of promotion-relevant arithmetic. `FLOOR_RESOLUTION` moved there for the same reason — the classifier needs it, and `test_mde == 0.0` is the too-narrow test the execution track already had to widen once: a null split over a constant difference returns 2.8e-17, so an `== 0.0` check goes inert on exactly the degenerate suites it exists for. Value unchanged; ledger row added. `ExecutionGateVerdict` also records the PREDECLARED `primary_criterion_index` and its `primary_mean_diff` — the paired difference on that criterion alone, which is what converts the blended figure back into the grader's own unit beside `dead_weight`. Computed over the rows the statistic actually paired, because the field is sold as that conversion and the identity holds only over one sample. The reviews changed real behaviour, not just prose: - `classify_confirm` accepted ANY pair of signs, and produced actively misleading answers for three reachable inputs: a candidate that lost on train and lost harder read REPRODUCED, a loss that shrank read SHRANK, and an effect that APPEARED where Stage B measured none read REPRODUCED. Stage C confirms a WIN; all of them are UNDECIDED now, plus a `math.isfinite` guard, since a NaN fell through to the most permissive rung. - The activation split check DROPPED its refusal whenever any pooled dir lacked provenance — `SplitProvenance.value` collapses to UNRECORDED when one dir is unreadable, so three dirs recording `train` beside one unreadable `run.json` classified over train rows with only a note. The shared `confirm_split_check` reads `recorded` directly; the regression test was mutation-verified against the chained form. - `primary_mean_diff` was never rendered while four surfaces said the block reported it. - "It never affects `promoted`" was false in three places: an over-range index refuses. - A refusal was printed twice per block. `holm_promote`'s rule applies here too. Phase 7 rides along, being prose the confirm gate's own scope depends on: `proposal-prompt.md` now names scripts and reference files as legitimate edit targets (with the prose→determinism hypothesis, and the three constraints that follow — `allowed_tools`, the whole-directory leak scan, activation being untouched), supplies the PASSING rows as what must not regress with the honest sample-and-mixed-evidence caveats, and points at the regression corpus before the edit rather than after the run. `outcome.yaml` gains the `Bash` worked example on its `allowed_tools` union rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…surfaces The bootstrap is seeded, so a p near the Holm threshold can land on either side of it depending on the draw — and a single verdict cannot say whether that happened. `gate_seed_stability` gates at three seeds and reports the agreement, the p per seed and the spread. **It returns no single `promoted` field, and that is the whole design.** Collapsing three disagreeing seeds into one verdict is exactly what the reading exists to prevent, so there is nothing to read that way: `render_seed_stability` says "UNSTABLE — promoted at 2/3 seeds. This is a coin flip, not a result" in those words, because "2/3" on its own reads like a result to anyone skimming. A separate function rather than a `seeds=` parameter on the gate. The parameter would have moved the cost and the rendered output of every existing call site and every pinned fixture; this way `activation_gate` is untouched — asserted on its signature — and the reading costs three bootstraps over rows already loaded, so zero extra agent runs. The skill says so where it offers it, since the obvious assumption is that it triples the round. `SeedStability` is a NamedTuple beside its producer on `RuleCeiling`'s stated precedent — computed and rendered, never persisted — and is exported from nowhere. The layering test's deferred-import set is three modules now, and every member of it is a NamedTuple: a new name appearing there is the signal to ask which category it belongs to. No execution twin, and the docstring says why rather than leaving it as an omission: that track's primary is an analytic paired *t*, deterministic given the rows, so a seed moves only the MDE and the guardrails — the function would report a spread of zero on the number that decides. Also finishing Phase 7's remaining surfaces, all of them consequences of scripts becoming a legal candidate shape: Step 12 diffs the whole skill DIRECTORY on the execution track (a `SKILL.md`-only diff renders a scripts-only candidate as no change at all); the method file's control-arm section says that a short body is no longer a proxy for "no instruction", so read the directory rather than the line count when deciding which arm is which; and Step 13's stop rule prints the cumulative CANDIDATE count beside the round count, because the patience is a budget in hypotheses and two rounds of four have tested eight while two rounds of one have tested two. Four prose sensors pin the paragraphs that would otherwise be one careless edit from vanishing — scripts as edit targets, the passing rows with their honest caveats, the candidate count, and the tree diff. Their absence is silent in every case: nothing fails, the proposer simply stops being told. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`CriterionResult` is a consumer contract — `task.json` is read by the evalboard and by anything pointed at a run tree — and CE030's doc-parity family covers `TaskDefinition` / `RunLimits` / `Dataset` / `SimulationConfig` only. So adding `weight` to the model and forgetting `docs/REPORT_SCHEMA.md` broke nothing and failed nothing; a reviewer caught it, which is exactly the kind of catch a guard should be making instead. `TestPersistedCriterionResultFieldsAreDocumented` asserts every base field is named in that section. Deliberately not a numbered rule: it is one derived assertion over one section rather than CE030's configurable model→guide machinery, and a number would imply a generality it does not have. Verified by mutation — removing `weight` from the section fails it. Two candidates deferred to `.claude/harness-candidates.md` with why neither is cheap: a python fence in a shipped SKILL.md being executable as written (the cheap `ast` version cannot tell a legitimate placeholder from an undefined local, and this is the SECOND fence defect to ship past every sensor, so the pair argues for building it properly), and a moved WATCHED constant leaving the schema doc's boundary paragraph misattributing it (a module check was written and then removed for being unfailable — the limitation is now stated in the test itself). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…four blind digests Six Mediums from the cross-phase pass, every one a seam between phases rather than a defect inside one. **The three magnitudes on an execution block were measured over three different row sets, and a comment claimed they were one.** `mean_diff` comes from `experiment.json`, `primary_mean_diff` from the on-disk results over the paired rows, `dead_weight` from the on-disk results over the intersection — so the conversion the block invites, `mean_diff ≈ primary × (1 − dead_weight)`, is exact only while all three coincide. Reproduced at 2.2× off with one row that `experiment.json` scored and disk does not carry. `_sample_divergence_note` now says so on the block, reporting the primary's USABLE count rather than the row-id list it was handed — the ghost row is IN that list and contributes nothing, which is the divergence itself. The identity test states its precondition and a second test pins the note's absence on a healthy block. **The primary-index refusal was recorded AFTER `_execution_diagnostics`, defeating the guards the previous phase added.** It produced exactly the contradiction those guards exist to prevent: a `NOT A RESULT — primary_criterion_index=7 selected no usable row` headline above notes reading "this is an ordinary negative result and not a measurement problem". Resolved before the ladder. **`gate_seed_stability` decided at a family of ONE while the round it is offered beside decided at `alpha/m`.** Measured: `3/3, STABLE` for a candidate a family of three rejects. It cannot reproduce Holm without every sibling's p at every seed, so it now says what it measures — "would promote at 3/3 seeds" plus a line naming the family size and sending the reader to the p spread. **The suite fingerprint was blind to five fields that decide what a score means**, sharpest of them `reference` — the answer key the judge criteria score against, while the module's own justification for hashing criteria whole is that the answer key is part of the instrument. Also `initial_prompt_file` (the prompt section hashed `null` for every file-backed suite, so the docstring's claim to cover the prompt was vacuous there), `expected_commands`, `pre_run`/`post_run` and `simulation`. Dumped with `include=`, which makes "the agent and sandbox blocks do not reach the digest" structural rather than a matter of which keys a loop reads. **The "51.2% dead weight by design" example was false in five prose surfaces.** The template's `file_check` is a GRADED outcome check with its own `mean` threshold; only the engagement criterion saturates by design, so the by-design figure is ~2.4%, and 51.2% is a property of a RUN in which the artifact check also saturated. Corrected everywhere, with the distinction stated. **`test_the_p_spread_is_none_below_two_measured_values` asserted the value it passed in** — the `>= 2` guard could be mutated to `>= 1` with the whole class green. It drives the real computation now, through the single-seed input that reaches the branch, and fails under that mutation. Lows: the byte-identical `family_resamples` line is `_family_resamples` at rank 1 (the duplication `_note_resolution_degraded` was put there to avoid); four statements about the deferred imports said "two NamedTuples" over four across three modules; the activation confirm de-duplicates its run dirs, since that track passes the same list for both arms and the note read "missing from 2 of 6" for three directories; the cost line derives its count from `len(seeds)`; one unused test parameter. Not fixed, and deliberately: five pre-existing defects the multi-model pass surfaced in adjacent untouched code — `_recorded_replicates`' handling of a malformed `replicate_index`, `search_compare` passing a whole cross-suite corpus, `rule_row_map`'s partial attribution, `render_headroom_ceilings`' GAP verdict under incomplete attribution, and `checker._resolve_refs`' reference exclusivity. `git blame` puts all five outside this change; they are listed as next steps rather than folded in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

uipreliga and others added 9 commits August 12, 2026 11:30

style: apply ruff format to the reachability lint assertion

fdae123

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs(harness): record the all-skipped-run-exits-0 gap found while add…

2dd5e14

…ing --split Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

uipreliga requested review from akshaylive, bai-uipath and tmatup as code owners August 12, 2026 21:36

uipreliga and others added 5 commits August 12, 2026 14:46

This comment was marked as outdated.

Sign in to view

uipreliga and others added 11 commits August 13, 2026 08:41

docs(harness): defer three gaps found reviewing the optimize-skill plan

278e4a6

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

uipreliga and others added 30 commits August 17, 2026 13:24

Navigation Menu

Uh oh!

feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced - #109

feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced#109
uipreliga wants to merge 168 commits into
mainfrom
feat/plugin-optimize-skill

uipreliga commented Aug 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

uipreliga commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

The feature

Why the two gates differ (worth a reviewer's eye)

Validation

Three bugs caught before shipping

The tutorial (docs/tutorials/08)

Notes for review

Late changes (after the first review pass)

Late changes (second pass): the execution track measured for real — and a criterion bug it uncovered

⚠️ Behaviour change reviewers should look at

What the run found

Corrections to claims made above

Also in this pass

Third pass — the review's P0/P1/P2 fixes, then a tutorial audit

Two behaviour changes to look at first

The rest

Tutorial audit

Notes for review

Uh oh!

github-actions Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review in Progress

Todo List

Uh oh!

This comment was marked as outdated.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

uipreliga commented Aug 12, 2026 •

edited

Loading

github-actions Bot commented Aug 12, 2026 •

edited

Loading