research: attention budget measured + Agent Loop Engineering rubric (25/28, gate FAIL)

[wjlgatech]

Two external sources, both of which assert without verifying. Neither is taken at its word.

A — @skills (arXiv:2608.12610, Yin et al., 12 Aug 2026)

The paper argues skills compete for "fewer than a hundred reliable auto-trigger slots per agent" and is explicit that this is not measured: "bounded by argument and by the literature rather than measured by us", naming the experiment as future work.

research/attention-budget/residency_audit.py — offline, deterministic, measures and proposes, never mutates. On a real 176-skill library: 1.76x over the argued bound; resident index 62,010 chars (~15.5k tokens) against 2.95M chars of bodies = 47.6x; 17 phantom entries (dangling symlinks that hold a name and load nothing), 25 duplicate names, 130/176 descriptions past the ~120-char guidance.

research/attention-budget/trigger_reliability.py — runs the paper's own named future work:

N installed	usable	top-1	95% CI
10	24	100.0%	86–100%
40	24	100.0%	86–100%
80	24	83.3%	64–93%
176	9	44.4%	19–73%

The break falls between 40 and 80 — below the argued 100, so the bound is directionally supported and, on this corpus, generous. Subject and probe-generator are deliberately different model families. Failed calls are excluded from the denominator, never scored as misses. Declared bias: probes derive from each target's description, so this is an optimistic upper bound. The N=176 cell rests on 9 trials and shows a collapse without locating one.

Two self-corrections, shipped as commits rather than quietly fixed

• 382759f — my frontmatter reader used ^description:\s*(.*)$, which captures the indicator of a YAML block scalar. Every description: | skill measured as a one-character description. It never failed; it returned a plausible wrong value. That made the audit numbers wrong (86.6x → 47.6x, 72 → 130 overlong) and corrupted the experiment's ground truth — three unrelated targets received the identical probe "Hey, can you pull". Those results were discarded, not published. Fixed in skillmeta.py, pinned by 10 tests including a regression guard.
• 5f910e0 — "~190 MCP tools" was eyeballed. Counted: 132. Retracted in place.

B — Agent Loop Engineering (webinar summary, 2026-08-20)

Sixteen sections of what a mature agent loop must have, verified against nothing. Encoded as a rubric:

• docs/rubrics/agent-loop-engineering.yml — the claims as data, single source
• scripts/audit_loop_rubric.py — probes them against shipped code and executed tests

Evidence is observed (a symbol at file:line, or a pytest node that actually ran). No evidence means no. A probe that cannot run is unmeasured: excluded from the rate and blocking.

The first run scored 23/23, and that was the bug. A rubric authored by someone who already knows the codebase, which then passes everything, has measured nothing. Rereading the source for claims chosen because I expected them to fail gives the honest score:

25/28 (89%) implemented · 3 declared gaps · 0 unmeasured · gate FAIL

Made true here: MemoryStore.success_rate() — section 14 asks for tokens, time, iterations and success rate; the first three were per-run and the fourth did not exist. In-flight runs are excluded from the denominator; blocked_safety counts as a failure, because a rate that forgave safety blocks would rise when the gate fires; returns None, never 0.0, when nothing has finished. 6 tests.

Still red, with a reason recorded beside the mark and no mechanism to convert a reason into a pass: ReAct interleaving and manager-worker (design divergences — ReAct lets the actor judge its own next step, which breaks maker≠checker), and human takeover of a running loop (a real gap, no plan).

Three declared gaps whose probes fail the moment the gap silently closes: typed failure taxonomy, branching search, span-level trace schema.

Evidence

606 tests pass, ruff clean. python3 scripts/audit_loop_rubric.py --gate exits 1 today, by design.

🤖 Generated with Claude Code