diff --git a/AGENTS.md b/AGENTS.md index 5b189db..55fdf3e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -96,7 +96,7 @@ protocol, so the refine engine is selectable: | `src/loopeng/router.py` | thin shim → `domains.REGISTRY.resolve`, adapts to legacy `LaneDecision` (U3; registry-backed plan-004 U11) | | `src/loopeng/domains/` | domain SDK: `Domain` plugin protocol + `DomainRegistry` (classify→resolve, supersedes router heuristics); `software.py` re-homes service/codebase lanes as registered domains. A new domain is a `register()`, never a controller/router edit (plan-004 U9/U11, KTD1/R11) | | `src/loopeng/domains/physical_ai/` | physical-AI-in-sim domain (plan-004 Phase B): `sim_judge.py` `SimJudge` referees a policy in sim over a **secret held-out** seed set → `Verdict`; `safety_profile.py` centralizes the CMDP cost gate (`derive_safety_ok`, KTD2). Sim is gated (skip-not-fail); reporting bound to "sim performance only" (R12). Adopt-actuator + registration land in U13 (U12) | -| `src/loopeng/memory/` | SQLite run history + trend/plateau/recurring queries (U2); `runs.finished` wall-clock; `iterations.score` continuous signal + score-aware `is_plateaued(on_score=)` (plan-004 U9/U10); concurrency-safe for parallel fan-out — writes serialized through one shared connection + `RLock` in WAL mode (plan-004 U16, R9); learnings portability (plan 2026-07-02 U1) — `portability.py` exports the learnings corpus as stable JSONL (`--redact` = cross-target `redact_specifics`) and imports it idempotently through `record_learning` (the single sanitize-on-write path), anchored to synthetic `imported`-status runs; surfaced as `loop-anything learnings export/import` | +| `src/loopeng/memory/` | SQLite run history + trend/plateau/recurring queries (U2); `runs.finished` wall-clock; `iterations.score` continuous signal + score-aware `is_plateaued(on_score=)` (plan-004 U9/U10); concurrency-safe for parallel fan-out — writes serialized through one shared connection + `RLock` in WAL mode (plan-004 U16, R9); learnings portability (plan 2026-07-02 U1) — `portability.py` exports the learnings corpus as stable JSONL (`--redact` = cross-target `redact_specifics`) and imports it idempotently through `record_learning` (the single sanitize-on-write path), anchored to synthetic `imported`-status runs; surfaced as `loop-anything learnings export/import`; `success_rate(target=None)` is the cross-run outcome metric — in-flight runs excluded from the denominator, `blocked_safety` counted as a failure, `None` (never `0.0`) when nothing has finished | | `src/loopeng/adapters/` | contracts, `safety.py` (subprocess/jail/env-prune), factory + judge shells (U4/U5), `compound_engineering.py` (`/ce-work` refiner), `llm_refiner.py` (claude-free fallback-chain refiner + `ChainedRefiner`: claude→LLM, infra-fail fall-through only, `last_refiner` provenance). `judge.py` `resolve_judge_adapter` — fail-closed, out-of-jail adapter discovery (refuses any adapter inside the maker's write tree so the referee stays immutable, 2026-06-18) | | `src/loopeng/bindings.py` | `build_loop_deps` — default judge/refiner/compounder from config + flags (`chain`/`claude`/`llm`); leaf module shared by `cli` `run` and `orchestration` fleet runner to avoid an import cycle (2026-06-18) | | `src/loopeng/adopt.py` | catalog tool adopter — venv-isolated, env-pruned, full-SHA-pinned (proof pipeline U1, KTD7) | @@ -104,6 +104,8 @@ protocol, so the refine engine is selectable: | `src/loopeng/proof.py` | `ProofPack` builder + `StoreBackedCompounder` (proof pipeline U3) | | `src/loopeng/loop/` | controller, convergence, brief, compound, `GitCheckpoint` (U6); `integrity.py` — maker≠checker / oracle≠checker / oracle≠maker / referee-immutability / held-out-disjoint assertions + human-confirm verification gate, all fail-closed (plan-004 U17, R6/R10, KTD6); `fork_card.py` + `resolver.py` — the Fork-Card decision channel: a build decision the spec didn't determine, resolved spec→oracle→escalate, reversed via existing rollback (plan 2026-06-17) | | `src/loopeng/autonomous/` | research report + autonomous runner; `run_refine_loop` (refine-only, proof pipeline U2); runs the U17 integrity preflight + gates `CONVERGED` via `RunResult.shippable` (`scheduled`/`confirmed`); `parallel.py` — worktree fan-out (`run_parallel`): one git worktree per target, bounded by `max_parallel`, crash-isolated, auto-cleaned (plan-004 U16, R9) | +| `docs/rubrics/` + `scripts/audit_loop_rubric.py` | The Agent Loop Engineering rubric as **data** (one entry per claim, single source) and the only program that reads it. Probes are `grep` (a symbol at `file:line`) or `pytest` (a node that is actually executed) — evidence is observed, never asserted. No evidence ⇒ fail; a probe that cannot run ⇒ `unmeasured`, excluded from the rate AND blocking; `expect: absent` items are declared gaps whose probes FAIL if the gap silently closes; `rationale` records why a red item stays red and can never convert it to a pass. `--gate` exits 1; `--rubric PATH` scores an archived version, and a rubric declaring `provenance:` has that banner rendered into its report so a reconstruction cannot look like a clean run. Current: 25/28, gate FAIL. | +| `research/attention-budget/` | Applies + measures the `@skills` residency principle (arXiv:2608.12610). `residency_audit.py` is offline/deterministic and never mutates a library; `trigger_reliability.py` measures top-1 routing vs installed-skill count using two disjoint model families, excludes failed API calls from the denominator, disk-caches every response, and reports "insufficient data" below `MIN_TRIALS` rather than a number. `skillmeta.py` is the shared frontmatter reader — a local regex here once returned a YAML block-scalar indicator (`\|`) as a description and silently corrupted both the audit and the experiment's ground truth. | | `src/loopeng/scheduler/` | `Heartbeat` cadence engine — durable `schedule_state`, due-calc, failure isolation, resume anchor; runner-agnostic (injected, KTD7). `tick` (sequential) + `tick_parallel` (fans due targets through `autonomous/parallel.py` into isolated worktrees, plan-004 U16). `loop-anything schedule add/list/remove/tick` (plan-004 U14, R7) | | `src/loopeng/orchestration/` | fleet orchestration layer (plan-006) — coordinates *many* self-improving loops under one goal ABOVE the per-target controller. `coordinator.run_fleet` runs items in topological waves over `autonomous/parallel.run_parallel` (cycles fail closed; non-converged deps block dependents; escalations PARK the fleet `awaiting_human`); `routing.py` pulls deps' outcomes into a dependent's brief via the U3 `upstream_context` seam; `escalation.py` routes only blocked/gated/stuck items to a human + `rebrief_item`; `spec.py`/`fleet_report.py` back the `loop-anything fleet` CLI. `default_fleet_runner` drives a real `run_refine_loop` per item inside its worktree (generate into the worktree, resolve an out-of-jail adapter, referee protected, `upstream_context` routed) — `fleet run` executes by default, `--dry-run` materializes only (2026-06-18). Per-item `target`/`goal`/`lane` live on the spec + `fleet_items` + `FleetItem`. Depends only on `run_parallel` + `RunResult` + the store — the `LoopController` is untouched (KTD1). `memory/fleet_state.py` holds the lifecycle enums + transition guard. | | `src/loopeng/demos/` | demo manifest/registry + result fixtures (validated; SSRF/traversal/secret guards) | diff --git a/CHANGELOG.md b/CHANGELOG.md index cf9e351..01cb341 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,34 @@ All notable changes to this project are documented here, following ## [Unreleased] ### Added +- **Agent Loop Engineering rubric — this engine scored against someone else's checklist + (25/28, gate FAIL).** A webinar summary listed sixteen sections of what a mature agent loop + must have and verified none of it against a running system. Encoded as data in + `docs/rubrics/agent-loop-engineering.yml`, scored by `scripts/audit_loop_rubric.py`, which is + the only consumer — so the published score cannot drift from the claims it scores. Evidence + is **observed** (a symbol at `file:line`, or a pytest node actually executed); no evidence + means no; a probe that cannot run is `unmeasured`, excluded from the rate **and** blocking. + `--gate` exits 1. **The first version scored 23/23 and that was the defect** — it was authored + by someone who had just read the codebase and contained no question he expected to fail. It is + kept at `docs/rubrics/archive/` as a labelled reconstruction so the claim is reproducible + (`--rubric …/agent-loop-engineering.v1.yml`), and the rubric's `provenance:` field is rendered + into any report, so a reconstruction cannot emit output that looks like a clean run. Three + items stay red with a `rationale` beside the mark and no mechanism to convert one into a pass; + three declared gaps carry probes that fail if the gap silently closes. +- **`MemoryStore.success_rate(target=None)` — cross-run task outcome.** Per-run token cost, wall + time and iteration count were already recorded; nothing ever divided. Two choices keep the + number honest: a run still in flight is **excluded from the denominator**, and `blocked_safety` + **counts as a failure**, so the metric cannot rise when the safety gate fires. Returns `None`, + never `0.0`, when nothing has finished — *not measured* must not read as *tried and failed*. + 6 tests in `tests/test_success_rate.py`. +- **`research/attention-budget/` — the `@skills` residency principle applied and measured.** + `residency_audit.py` scores a real skill library against arXiv:2608.12610's three-tier model + (offline, deterministic, measures and proposes, never mutates). `trigger_reliability.py` runs + the experiment that paper explicitly does **not** run: top-1 routing across a 175-skill corpus + measured at 100% (N=10), 100% (N=40), 83.3% (N=80), 44.4% (N=176) — the break falling below + the paper's argued 100-slot bound. Declared bias, confidence intervals, and two + self-corrections (a YAML block-scalar parse bug that invalidated the first numbers, and a + retracted eyeball estimate) are recorded in that directory's README rather than quietly fixed. - **The run contract (`loop.yaml`) — a reviewable order ticket for a loop** — run inputs (target, goal, lane, ten budget knobs) existed only as CLI flags: un-diffable, unreviewable, and impossible to attach to the proof pack the run produced. Ships diff --git a/README.md b/README.md index c75593a..5cae250 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ reality, and refactors until the grade stops climbing — then tells you what it [![CI](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml/badge.svg)](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml) [![loop-anything-hub](https://img.shields.io/badge/loop--anything--hub-live-brightgreen)](https://wjlgatech.github.io/loop-engineering-anything/) -[![Tests](https://img.shields.io/badge/tests-315%20passing-brightgreen)](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml) +[![Tests](https://img.shields.io/badge/tests-606%20passing-brightgreen)](https://github.com/wjlgatech/loop-engineering-anything/actions/workflows/ci.yml) [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org/) [![Units](https://img.shields.io/badge/plan-6%2F8%20units-blue)](docs/plans/) [![License](https://img.shields.io/badge/license-MIT-green)](#-license) @@ -269,6 +269,63 @@ Three rules keep it from becoming decoration `evidence.required` is checked against the run's **real** proof pack and exits non-zero on a gap — so declaring evidence is a claim the run has to satisfy. +### 🧮 Scored against someone else's checklist — and it fails + +A webinar summary listed sixteen things a mature agent loop must have, and verified none of +them against a running system. That is a rubric wearing an essay's clothes, so it is encoded +as one: [`docs/rubrics/agent-loop-engineering.yml`](docs/rubrics/agent-loop-engineering.yml) +holds the claims as **data**, and [`scripts/audit_loop_rubric.py`](scripts/audit_loop_rubric.py) +is the only thing that reads them, so the published score cannot drift from the claims it scores. + +```bash +python3 scripts/audit_loop_rubric.py # score this engine +python3 scripts/audit_loop_rubric.py --gate # exit 1 unless every item passes +``` + +``` +**Conformance: 25/28 (89%)** verifiable claims implemented · 3 declared gaps · 0 unmeasured · gate **FAIL** +``` + +Evidence is **observed** — a symbol at a real `file:line`, or a pytest node that is actually +executed. No evidence means no. A probe that cannot run is `unmeasured`: excluded from the rate +**and** blocking, because "I could not look" is not "it works". + +**The first version of this rubric scored 23/23, and that was the bug.** It was written by +someone who had just spent an hour in this codebase, and it contained no question he expected +to fail — an assessment that cannot fail its author is a mirror. That run is kept, honestly +labelled a reconstruction, so the claim is reproducible rather than asserted: + +```bash +python3 scripts/audit_loop_rubric.py --rubric docs/rubrics/archive/agent-loop-engineering.v1.yml +# 23/23 (100%) · gate PASS ← the score that measured nothing +``` + +Three items stay red with a `rationale` recorded beside the mark and **no mechanism to turn a +rationale into a pass** (ReAct interleaving and manager-worker are deliberate divergences; +human takeover of a running loop is a real gap). Three further **declared gaps** — typed +failure taxonomy, branching search, span-level trace schema — carry probes that *fail the +moment the gap silently closes*, so the limitation list cannot rot. + +Full output: [`docs/rubrics/AUDIT.md`](docs/rubrics/AUDIT.md). + +### 📊 Did it actually work? — cross-run success rate + +Per-run cost, wall time and iteration count were always recorded; nothing ever divided. +`MemoryStore.success_rate(target=None)` closes that, and two choices decide whether the number +is honest: a run **still in flight is excluded from the denominator**, and a run halted by the +safety gate **counts as a failure** — otherwise the metric would rise when the safety gate +fires. It returns `None`, never `0.0`, when nothing has finished: *not measured* must not read +as *tried and failed*. + +### 🔬 Research — the attention budget, measured + +[`research/attention-budget/`](research/attention-budget/) applies the `@skills` protocol +(arXiv:2608.12610) to a real skill library and then measures the quantity that paper explicitly +does **not** measure. Top-1 routing against a 175-skill corpus: **100% at N=10 and N=40, 83% at +N=80, 44% at N=176** — the break falls below the paper's argued 100-slot bound. Limits, bias and +two self-corrections (a YAML block-scalar parse bug that made the first numbers wrong) are +recorded in that directory's README rather than quietly fixed. + --- ## ⚙️ How it works @@ -381,6 +438,7 @@ loop-engineering-anything/ │ ├── adopt.py # catalog tool adopter — venv-isolated, env-pruned, full-SHA pin │ ├── proof.py # ProofPack builder + store-backed compounder │ ├── router.py # target → lane classification +│ ├── memory/store.py # + success_rate(): cross-run outcome, safety blocks count as failures │ ├── adapters/ │ │ ├── base.py # Verdict / GenerateResult + Judge/Refiner/Compounder/Checkpoint protocols │ │ ├── safety.py # shell=False exec, metachar rejection, workspace jail @@ -390,8 +448,11 @@ loop-engineering-anything/ │ ├── memory/ # SQLite store + trend/plateau/recurring queries │ ├── loop/ # controller state machine, convergence, brief, compound, GitCheckpoint │ └── autonomous/ # research report + autonomous runner +├── docs/rubrics/ # the Agent Loop Engineering rubric as data + its generated audit +├── research/attention-budget/ # the @skills residency audit + the trigger-reliability experiment +├── scripts/audit_loop_rubric.py # scores this engine against the rubric; --gate exits 1 ├── skills/loop-anything/ # the /loop-anything agent skill -├── tests/ # 600 tests — loop dynamics validated against recorded verdicts +├── tests/ # 606 tests — loop dynamics validated against recorded verdicts └── docs/plans/ # the implementation plan ```