wjlgatech · wjlgatech · Aug 21, 2026 · Aug 21, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -96,14 +96,16 @@ protocol, so the refine engine is selectable:
 | `src/loopeng/router.py` | thin shim → `domains.REGISTRY.resolve`, adapts to legacy `LaneDecision` (U3; registry-backed plan-004 U11) |
 | `src/loopeng/domains/` | domain SDK: `Domain` plugin protocol + `DomainRegistry` (classify→resolve, supersedes router heuristics); `software.py` re-homes service/codebase lanes as registered domains. A new domain is a `register()`, never a controller/router edit (plan-004 U9/U11, KTD1/R11) |
 | `src/loopeng/domains/physical_ai/` | physical-AI-in-sim domain (plan-004 Phase B): `sim_judge.py` `SimJudge` referees a policy in sim over a **secret held-out** seed set → `Verdict`; `safety_profile.py` centralizes the CMDP cost gate (`derive_safety_ok`, KTD2). Sim is gated (skip-not-fail); reporting bound to "sim performance only" (R12). Adopt-actuator + registration land in U13 (U12) |
-| `src/loopeng/memory/` | SQLite run history + trend/plateau/recurring queries (U2); `runs.finished` wall-clock; `iterations.score` continuous signal + score-aware `is_plateaued(on_score=)` (plan-004 U9/U10); concurrency-safe for parallel fan-out — writes serialized through one shared connection + `RLock` in WAL mode (plan-004 U16, R9); learnings portability (plan 2026-07-02 U1) — `portability.py` exports the learnings corpus as stable JSONL (`--redact` = cross-target `redact_specifics`) and imports it idempotently through `record_learning` (the single sanitize-on-write path), anchored to synthetic `imported`-status runs; surfaced as `loop-anything learnings export/import` |
+| `src/loopeng/memory/` | SQLite run history + trend/plateau/recurring queries (U2); `runs.finished` wall-clock; `iterations.score` continuous signal + score-aware `is_plateaued(on_score=)` (plan-004 U9/U10); concurrency-safe for parallel fan-out — writes serialized through one shared connection + `RLock` in WAL mode (plan-004 U16, R9); learnings portability (plan 2026-07-02 U1) — `portability.py` exports the learnings corpus as stable JSONL (`--redact` = cross-target `redact_specifics`) and imports it idempotently through `record_learning` (the single sanitize-on-write path), anchored to synthetic `imported`-status runs; surfaced as `loop-anything learnings export/import`; `success_rate(target=None)` is the cross-run outcome metric — in-flight runs excluded from the denominator, `blocked_safety` counted as a failure, `None` (never `0.0`) when nothing has finished |
 | `src/loopeng/adapters/` | contracts, `safety.py` (subprocess/jail/env-prune), factory + judge shells (U4/U5), `compound_engineering.py` (`/ce-work` refiner), `llm_refiner.py` (claude-free fallback-chain refiner + `ChainedRefiner`: claude→LLM, infra-fail fall-through only, `last_refiner` provenance). `judge.py` `resolve_judge_adapter` — fail-closed, out-of-jail adapter discovery (refuses any adapter inside the maker's write tree so the referee stays immutable, 2026-06-18) |
 | `src/loopeng/bindings.py` | `build_loop_deps` — default judge/refiner/compounder from config + flags (`chain`/`claude`/`llm`); leaf module shared by `cli` `run` and `orchestration` fleet runner to avoid an import cycle (2026-06-18) |
 | `src/loopeng/adopt.py` | catalog tool adopter — venv-isolated, env-pruned, full-SHA-pinned (proof pipeline U1, KTD7) |
 | `src/loopeng/connectors/` | actuator layer — `Connector` protocol (structured `act(payload)`, never shell-interpolated) + install/credential isolation boundary: strict allowlisted `env=` (`minimal_env`), full-SHA pin, install outside the worktree, credentials by name only; one reference connector (plan-004 U15, KTD8/R8). Optional/injected — the controller never imports it (KTD7) |
 | `src/loopeng/proof.py` | `ProofPack` builder + `StoreBackedCompounder` (proof pipeline U3) |
 | `src/loopeng/loop/` | controller, convergence, brief, compound, `GitCheckpoint` (U6); `integrity.py` — maker≠checker / oracle≠checker / oracle≠maker / referee-immutability / held-out-disjoint assertions + human-confirm verification gate, all fail-closed (plan-004 U17, R6/R10, KTD6); `fork_card.py` + `resolver.py` — the Fork-Card decision channel: a build decision the spec didn't determine, resolved spec→oracle→escalate, reversed via existing rollback (plan 2026-06-17) |
 | `src/loopeng/autonomous/` | research report + autonomous runner; `run_refine_loop` (refine-only, proof pipeline U2); runs the U17 integrity preflight + gates `CONVERGED` via `RunResult.shippable` (`scheduled`/`confirmed`); `parallel.py` — worktree fan-out (`run_parallel`): one git worktree per target, bounded by `max_parallel`, crash-isolated, auto-cleaned (plan-004 U16, R9) |
+| `docs/rubrics/` + `scripts/audit_loop_rubric.py` | The Agent Loop Engineering rubric as **data** (one entry per claim, single source) and the only program that reads it. Probes are `grep` (a symbol at `file:line`) or `pytest` (a node that is actually executed) — evidence is observed, never asserted. No evidence ⇒ fail; a probe that cannot run ⇒ `unmeasured`, excluded from the rate AND blocking; `expect: absent` items are declared gaps whose probes FAIL if the gap silently closes; `rationale` records why a red item stays red and can never convert it to a pass. `--gate` exits 1; `--rubric PATH` scores an archived version, and a rubric declaring `provenance:` has that banner rendered into its report so a reconstruction cannot look like a clean run. Current: 25/28, gate FAIL. |
+| `research/attention-budget/` | Applies + measures the `@skills` residency principle (arXiv:2608.12610). `residency_audit.py` is offline/deterministic and never mutates a library; `trigger_reliability.py` measures top-1 routing vs installed-skill count using two disjoint model families, excludes failed API calls from the denominator, disk-caches every response, and reports "insufficient data" below `MIN_TRIALS` rather than a number. `skillmeta.py` is the shared frontmatter reader — a local regex here once returned a YAML block-scalar indicator (`\|`) as a description and silently corrupted both the audit and the experiment's ground truth. |
 | `src/loopeng/scheduler/` | `Heartbeat` cadence engine — durable `schedule_state`, due-calc, failure isolation, resume anchor; runner-agnostic (injected, KTD7). `tick` (sequential) + `tick_parallel` (fans due targets through `autonomous/parallel.py` into isolated worktrees, plan-004 U16). `loop-anything schedule add/list/remove/tick` (plan-004 U14, R7) |
 | `src/loopeng/orchestration/` | fleet orchestration layer (plan-006) — coordinates *many* self-improving loops under one goal ABOVE the per-target controller. `coordinator.run_fleet` runs items in topological waves over `autonomous/parallel.run_parallel` (cycles fail closed; non-converged deps block dependents; escalations PARK the fleet `awaiting_human`); `routing.py` pulls deps' outcomes into a dependent's brief via the U3 `upstream_context` seam; `escalation.py` routes only blocked/gated/stuck items to a human + `rebrief_item`; `spec.py`/`fleet_report.py` back the `loop-anything fleet` CLI. `default_fleet_runner` drives a real `run_refine_loop` per item inside its worktree (generate into the worktree, resolve an out-of-jail adapter, referee protected, `upstream_context` routed) — `fleet run` executes by default, `--dry-run` materializes only (2026-06-18). Per-item `target`/`goal`/`lane` live on the spec + `fleet_items` + `FleetItem`. Depends only on `run_parallel` + `RunResult` + the store — the `LoopController` is untouched (KTD1). `memory/fleet_state.py` holds the lifecycle enums + transition guard. |
 | `src/loopeng/demos/` | demo manifest/registry + result fixtures (validated; SSRF/traversal/secret guards) |

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,34 @@ All notable changes to this project are documented here, following
 ## [Unreleased]
 
 ### Added
+- **Agent Loop Engineering rubric — this engine scored against someone else's checklist
+  (25/28, gate FAIL).** A webinar summary listed sixteen sections of what a mature agent loop
+  must have and verified none of it against a running system. Encoded as data in
+  `docs/rubrics/agent-loop-engineering.yml`, scored by `scripts/audit_loop_rubric.py`, which is
+  the only consumer — so the published score cannot drift from the claims it scores. Evidence
+  is **observed** (a symbol at `file:line`, or a pytest node actually executed); no evidence
+  means no; a probe that cannot run is `unmeasured`, excluded from the rate **and** blocking.
+  `--gate` exits 1. **The first version scored 23/23 and that was the defect** — it was authored
+  by someone who had just read the codebase and contained no question he expected to fail. It is
+  kept at `docs/rubrics/archive/` as a labelled reconstruction so the claim is reproducible
+  (`--rubric …/agent-loop-engineering.v1.yml`), and the rubric's `provenance:` field is rendered
+  into any report, so a reconstruction cannot emit output that looks like a clean run. Three
+  items stay red with a `rationale` beside the mark and no mechanism to convert one into a pass;
+  three declared gaps carry probes that fail if the gap silently closes.
+- **`MemoryStore.success_rate(target=None)` — cross-run task outcome.** Per-run token cost, wall
+  time and iteration count were already recorded; nothing ever divided. Two choices keep the
+  number honest: a run still in flight is **excluded from the denominator**, and `blocked_safety`
+  **counts as a failure**, so the metric cannot rise when the safety gate fires. Returns `None`,
+  never `0.0`, when nothing has finished — *not measured* must not read as *tried and failed*.
+  6 tests in `tests/test_success_rate.py`.
+- **`research/attention-budget/` — the `@skills` residency principle applied and measured.**
+  `residency_audit.py` scores a real skill library against arXiv:2608.12610's three-tier model
+  (offline, deterministic, measures and proposes, never mutates). `trigger_reliability.py` runs
+  the experiment that paper explicitly does **not** run: top-1 routing across a 175-skill corpus
+  measured at 100% (N=10), 100% (N=40), 83.3% (N=80), 44.4% (N=176) — the break falling below
+  the paper's argued 100-slot bound. Declared bias, confidence intervals, and two
+  self-corrections (a YAML block-scalar parse bug that invalidated the first numbers, and a
+  retracted eyeball estimate) are recorded in that directory's README rather than quietly fixed.
 - **The run contract (`loop.yaml`) — a reviewable order ticket for a loop** — run inputs
   (target, goal, lane, ten budget knobs) existed only as CLI flags: un-diffable,
   unreviewable, and impossible to attach to the proof pack the run produced. Ships

diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ reality, and refactors until the grade stops climbing — then tells you what it
 
 [![CI](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml/badge.svg)](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml)
 [![loop-anything-hub](https://img.shields.io/badge/loop--anything--hub-live-brightgreen)](https://wjlgatech.github.io/loop-engineering-anything/)
-[![Tests](https://img.shields.io/badge/tests-315%20passing-brightgreen)](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml)
+[![Tests](https://img.shields.io/badge/tests-606%20passing-brightgreen)](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml)
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org/)
 [![Units](https://img.shields.io/badge/plan-6%2F8%20units-blue)](docs/plans/)
 [![License](https://img.shields.io/badge/license-MIT-green)](#-license)
@@ -269,6 +269,63 @@ Three rules keep it from becoming decoration
 `evidence.required` is checked against the run's **real** proof pack and exits non-zero on
 a gap — so declaring evidence is a claim the run has to satisfy.
 
+### 🧮 Scored against someone else's checklist — and it fails
+
+A webinar summary listed sixteen things a mature agent loop must have, and verified none of
+them against a running system. That is a rubric wearing an essay's clothes, so it is encoded
+as one: [`docs/rubrics/agent-loop-engineering.yml`](docs/rubrics/agent-loop-engineering.yml)
+holds the claims as **data**, and [`scripts/audit_loop_rubric.py`](scripts/audit_loop_rubric.py)
+is the only thing that reads them, so the published score cannot drift from the claims it scores.
+
+```bash
+python3 scripts/audit_loop_rubric.py                 # score this engine
+python3 scripts/audit_loop_rubric.py --gate          # exit 1 unless every item passes
+```
+
+```
+**Conformance: 25/28 (89%)** verifiable claims implemented · 3 declared gaps · 0 unmeasured · gate **FAIL**
+```
+
+Evidence is **observed** — a symbol at a real `file:line`, or a pytest node that is actually
+executed. No evidence means no. A probe that cannot run is `unmeasured`: excluded from the rate
+**and** blocking, because "I could not look" is not "it works".
+
+**The first version of this rubric scored 23/23, and that was the bug.** It was written by
+someone who had just spent an hour in this codebase, and it contained no question he expected
+to fail — an assessment that cannot fail its author is a mirror. That run is kept, honestly
+labelled a reconstruction, so the claim is reproducible rather than asserted:
+
+```bash
+python3 scripts/audit_loop_rubric.py --rubric docs/rubrics/archive/agent-loop-engineering.v1.yml
+# 23/23 (100%) · gate PASS   ← the score that measured nothing
+```
+
+Three items stay red with a `rationale` recorded beside the mark and **no mechanism to turn a
+rationale into a pass** (ReAct interleaving and manager-worker are deliberate divergences;
+human takeover of a running loop is a real gap). Three further **declared gaps** — typed
+failure taxonomy, branching search, span-level trace schema — carry probes that *fail the
+moment the gap silently closes*, so the limitation list cannot rot.
+
+Full output: [`docs/rubrics/AUDIT.md`](docs/rubrics/AUDIT.md).
+
+### 📊 Did it actually work? — cross-run success rate
+
+Per-run cost, wall time and iteration count were always recorded; nothing ever divided.
+`MemoryStore.success_rate(target=None)` closes that, and two choices decide whether the number
+is honest: a run **still in flight is excluded from the denominator**, and a run halted by the
+safety gate **counts as a failure** — otherwise the metric would rise when the safety gate
+fires. It returns `None`, never `0.0`, when nothing has finished: *not measured* must not read
+as *tried and failed*.
+
+### 🔬 Research — the attention budget, measured
+
+[`research/attention-budget/`](research/attention-budget/) applies the `@skills` protocol
+(arXiv:2608.12610) to a real skill library and then measures the quantity that paper explicitly
+does **not** measure. Top-1 routing against a 175-skill corpus: **100% at N=10 and N=40, 83% at
+N=80, 44% at N=176** — the break falls below the paper's argued 100-slot bound. Limits, bias and
+two self-corrections (a YAML block-scalar parse bug that made the first numbers wrong) are
+recorded in that directory's README rather than quietly fixed.
+
 ---
 
 ## ⚙️ How it works
@@ -381,6 +438,7 @@ loop-engineering-anything/
 │   ├── adopt.py               # catalog tool adopter — venv-isolated, env-pruned, full-SHA pin
 │   ├── proof.py               # ProofPack builder + store-backed compounder
 │   ├── router.py              # target → lane classification
+│   ├── memory/store.py        # + success_rate(): cross-run outcome, safety blocks count as failures
 │   ├── adapters/
 │   │   ├── base.py            # Verdict / GenerateResult + Judge/Refiner/Compounder/Checkpoint protocols
 │   │   ├── safety.py          # shell=False exec, metachar rejection, workspace jail
@@ -390,8 +448,11 @@ loop-engineering-anything/
 │   ├── memory/                # SQLite store + trend/plateau/recurring queries
 │   ├── loop/                  # controller state machine, convergence, brief, compound, GitCheckpoint
 │   └── autonomous/            # research report + autonomous runner
+├── docs/rubrics/              # the Agent Loop Engineering rubric as data + its generated audit
+├── research/attention-budget/ # the @skills residency audit + the trigger-reliability experiment
+├── scripts/audit_loop_rubric.py  # scores this engine against the rubric; --gate exits 1
 ├── skills/loop-anything/      # the /loop-anything agent skill
-├── tests/                     # 600 tests — loop dynamics validated against recorded verdicts
+├── tests/                     # 606 tests — loop dynamics validated against recorded verdicts
 └── docs/plans/                # the implementation plan
 ```