|
| 1 | +# ADR 0004: threshold calibration returns "insufficient data" - ships measurement, not new numbers |
| 2 | + |
| 3 | +- Status: proposed |
| 4 | +- Date: 2026-08-09 |
| 5 | +- Source spec: Jira SW-31 (`FEAT-threshold-recalibration`) |
| 6 | +- Relates to: SW-13 (ADR 0002, Gate Complexity thresholds); SW-10 / SW-16 (metrics log and |
| 7 | + `/sd:status`); SW-30 (second example corpus) |
| 8 | +- Supersedes: none |
| 9 | + |
| 10 | +## Context |
| 11 | + |
| 12 | +SW-31 asks for a repeatable calibration pass: turn accumulated `.specs/_metrics/events.jsonl` and |
| 13 | +spec-artifact data into evidence for or against each threshold the engine enforces - Gate |
| 14 | +Complexity (tasks > 8, layers > 2, files > 8, from ADR 0002), `retroStaleMinutes` (30), |
| 15 | +`debounceMinutes` (10), `maxLessons` (3), `metrics.maxSizeKb` (1024), and the perf gate's 5% noise |
| 16 | +floor - plus a standing ritual so calibration happens again. |
| 17 | + |
| 18 | +Running that pass against this repo's actual state surfaced two things the ticket's own text |
| 19 | +didn't anticipate: |
| 20 | + |
| 21 | +1. **The corpus is thinner than "n=1 risk" implies.** specwright does not dogfood itself - there is |
| 22 | + no `.specs/` directory in this repo at all. `examples/fixture-project`, the second corpus SW-31 |
| 23 | + was waiting on (SW-30), has exactly **one** closed spec (`FEAT-todo-priority`) and no |
| 24 | + `events.jsonl` history. Real accumulated data is n=1, not a "canyon" or a "cluster" - there is |
| 25 | + nothing to fit a distribution to yet. |
| 26 | +2. **Gate Complexity trips were never recorded.** The event schema |
| 27 | + (`docs/architecture.md`, "Event log") only ever emitted `gate: verify|protected|code-edit`. |
| 28 | + Gate Complexity (ADR 0002) is decided as model-executed prose inside `/sd:feature` Phase 3 Gate |
| 29 | + 2 - ADR 0002 itself names this an accepted, unresolved limitation ("no script... exercises the |
| 30 | + threshold arithmetic... automatically"). Before this ADR, there was no path to ever answering |
| 31 | + "what is the trip rate" from measured data, regardless of corpus size. |
| 32 | + |
| 33 | +Per ADR 0002's own "Scope declined" precedent, this ADR ships the machinery to close gap (2) |
| 34 | +**partially** - split detection only, not full trip-rate - and accepts gap (1) as the honest |
| 35 | +current state rather than manufacturing evidence from a single spec ("Re-fitting to the same five |
| 36 | +specs would launder a guess as data" - SW-31's own note, which applies at n=1 even more directly |
| 37 | +than the n=5 case it was written about). |
| 38 | + |
| 39 | +## Decision |
| 40 | + |
| 41 | +1. **New inferred metric: `gate:"complexity"` / `decision:"split"`.** `spec-gate` |
| 42 | + (`hooks/bash/spec-gate.sh`, `hooks/powershell/spec-gate.ps1`) now watches every `index.md` edit |
| 43 | + for the structural trace a completed Gate Complexity split leaves behind: a `FEAT-X` row newly |
| 44 | + transitioning to `archived` alongside any `FEAT-X-<slug>` row already registered (on disk or in |
| 45 | + the same pending edit), per `commands/feature.md`'s Face B "approve split" steps. This is |
| 46 | + observational only, computed from data the workflow already writes - no change to |
| 47 | + `commands/feature.md`'s Gate 2 (a HARD gate) or `agents/spec-architect.md` was made or is |
| 48 | + needed. It is emitted **only when the edit is actually allowed through**, never on a `block` |
| 49 | + exit - a denied edit never reaches disk, so a detected pattern inside it did not really happen. |
| 50 | + Under the default config, `.specs/index.md` is itself listed in `paths.protected` |
| 51 | + (`templates/project-config.template.json`), so most direct `index.md` edits are already blocked |
| 52 | + before this metric ever gets a chance to fire (`docs/architecture.md`'s existing note on the |
| 53 | + `decision` field: "Most direct index edits are blocked by `paths.protected`, so `block` is the |
| 54 | + common case"). That is a pre-existing, documented property of `spec-gate`, not something this |
| 55 | + ADR introduces or changes - but it does mean the split count will under-count real splits on |
| 56 | + any project that leaves `index.md` protected, which is the default. Fixing that tension (how |
| 57 | + `/sd:feature`'s own Gate 2 writes are meant to reach a protected `index.md` at all) is out of |
| 58 | + scope here; `commands/spec.md` already carries the same open tension for the `done` transition |
| 59 | + via its `verifyGate` carve-out. |
| 60 | +2. **`/sd:status --calibration`.** A new optional view (`commands/status.md`, Phase 3b) reports |
| 61 | + task/layer/file distributions read from spec artifacts, plus the `gate:"complexity"`/`split` |
| 62 | + count from `events.jsonl`. The default `/sd:status` invocation's read contract is unchanged. |
| 63 | +3. **The five non-Gate-Complexity thresholds are marked as judgement calls**, not measured values, |
| 64 | + directly in `templates/project-config.template.json` (`_retroStaleMinutes_use`, |
| 65 | + `_debounceMinutes_use`, `_maxLessons_use`, matching the existing `_maxSizeKb_use` caveat). Gate |
| 66 | + Complexity is left as-is; ADR 0002 already recorded its measured basis (the SW-13 corpus trace), |
| 67 | + which nothing in this ADR revisits or invalidates. |
| 68 | +4. **Calibration verdict, this run: insufficient data, for every threshold.** n=1 closed spec |
| 69 | + across both corpora, zero real `gate:"complexity"` events (the metric only exists as of this |
| 70 | + ADR), zero real `events.jsonl` history outside test fixtures. No threshold changes size on this |
| 71 | + run - per SW-31's own acceptance criterion, this is the honest and expected outcome at this |
| 72 | + corpus size, not a failure of the calibration pass. |
| 73 | +5. **CONTRIBUTING names the re-calibration trigger**: every 20 closed specs, or each minor release, |
| 74 | + whichever comes first (SW-31's own proposed cadence). The next run of `/sd:status --calibration` |
| 75 | + is the mechanism that answers whether that bar has been met. |
| 76 | + |
| 77 | +## Consequences |
| 78 | + |
| 79 | +**Positive.** Real calibration becomes possible going forward without re-opening this ticket - the |
| 80 | +next `/sd:status --calibration` run after real specs accumulate reads live data instead of nothing. |
| 81 | +The judgement-call caveats make future readers of `project-config.template.json` unable to mistake |
| 82 | +an untuned default for a measured one. |
| 83 | + |
| 84 | +**Negative.** `gate:"complexity"`/`split` cannot detect a bare trip (Face A vs. Face B "no-split" |
| 85 | +look identical in `index.md`) - see "Scope declined" below. The split-detection heuristic itself is |
| 86 | +best-effort: two unrelated specs that happen to share an id prefix (`FEAT-auth` / `FEAT-auth-v2`) |
| 87 | +would misread as parent/child. Acceptable for an observational, non-gate-affecting metric, same |
| 88 | +class of limitation the file already accepts elsewhere (Rule 0's bundled-edit limitation). |
| 89 | +Recording only on an allowed edit (see Decision 1) means the metric under-counts on any project |
| 90 | +that leaves `index.md` protected by default - it will report fewer splits than actually occurred, |
| 91 | +never more; a project relying on this signal needs to make the archive-plus-child edit reachable. |
| 92 | + |
| 93 | +**Scope declined.** Full trip-rate instrumentation (recording every Gate 2 resolution, not only |
| 94 | +completed splits) is **not** built here. The only way to observe it directly is a marker the |
| 95 | +architect or the `/sd:feature` command writes at Gate 2 resolution time - and making a HARD gate's |
| 96 | +prose responsible for reliably feeding a metrics pipeline would break the invariant |
| 97 | +`docs/architecture.md` states plainly: "`spec-gate` and `subagent-retro` are the hooks that |
| 98 | +record." A future spec that wants full trip-rate should design the instrumentation around a |
| 99 | +deterministic, hook-observable signal (or accept a model-authored marker as an explicit, separate |
| 100 | +trade-off) rather than retrofitting it into this "cheap ticket." |
| 101 | + |
| 102 | +## Follow-up |
| 103 | + |
| 104 | +Re-run this calibration once `.specs/index.md` (or `examples/fixture-project/.specs/index.md`) |
| 105 | +records at least 20 closed specs, or at the next minor release - whichever comes first, per |
| 106 | +CONTRIBUTING. A future ADR should supersede this one with the first real verdict. |
0 commit comments