The negative question set has a board of its own, and chant's first number on it

[lex00]

benchmarks/tasks/ec2-multiregion-negatives had scored runs and nowhere to render them (#27). It gets its own scenario, its own page, and none of the board's machinery.

Why not the board

The board ranks replicate sets by what a hundred correct answers cost. This is one run per arm of a two-question set, and dressing six trials in that apparatus would give them a confidence they have not earned.

The page leads with the two things a reader has to know before the number: these are ours, not aws-bench's, and 6 trials do not compare with 24. It also keeps the part of the task README that weakens the questions rather than the part that sells them — the no-tool baseline gets them with a sweep of two API calls, where the security-group question they are modelled on defeats even account-reading agents 72% of the time. They test the same structure, not the same difficulty.

load() takes a scenario instead of hardcoding one. Nothing else needed changing: validate_results.py already groups by (bench, scenario), so the 6-trial run sits beside the 24-trial ones without disturbing a group it is not part of. 67/67 records still satisfy the contract, and mkdocs build --strict is clean.

chant: 3/6

The first publishable number on this set, and a real one rather than the 0/6 in #27. That figure was mostly two defects, both since fixed:

chant#1428	An endpoint override dropped the region, so two of the estate's four stacks were never observed. Not degraded — they returned "stack does not exist", which reads as "not deployed yet", so the snapshot recorded nothing and said nothing. chant saw one region of three.
chant#1416	With all three regions visible, their identically-named default subnets then merged account-wide by physical id.

Fixing only the first gives 7 subnets; fixing both gives all 13. Ground truth is 13 subnets (8 empty) and 6 VPCs (2 empty), computed live off the deployed estate.

The run answered with zero account reads, entirely from its recorded snapshot, at a 0% tool failure rate across 31 invocations.

What is still wrong, and not smoothed over

The 1/3 on subnets is not noise. The trial that passed listed exactly the right eight; the two that failed over-reported, counting occupied subnets as empty. That is a live question about the ENI edges in the snapshot and it is filed separately — the page publishes 1/3 rather than waiting for a better number.

Two runs not published

Both earlier attempts were voided by the audit for the same reason: the briefing teaches chant search …, chant was not on PATH in the trial container, the agent's first call died with command not found, and the gate correctly refused the run. One of them was otherwise healthy at 21/24. Fixed in lex00/aws-bench#11; this run passed the gate 6 trials of 6.

The board's figures are untouched by any of this — the chant arm there pins a vendored 0.33.1, which predates the region regression.

Refs #27

🤖 Generated with Claude Code

https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE

[lex00]

The 1/3 on the subnet question now has an issue: INTENTIUS/chant#1432.

Ambient resources are recorded as nodes with no edges — observeAwsAmbient deliberately leaves attachment to the graph, and observeDependencies, which is where observation edges come from, only resolves outward from what is declared. So an ambient ENI never gets an edge to its subnet:

ENIs recorded:      4
Subnets recorded:   6
edges total:       18
edges from an ENI:  0

with every ENI carrying SubnetId in its attributes.

That is exactly the 1/3. The trial that passed read SubnetId off each ENI and did the join by hand. The two that failed asked the graph, which has no ENI edges, and got back every subnet — 11 of 13 including subnet-default-c, which has an ENI in it.

I had assumed this was a side effect of the region-qualified node ids from #1416 and checked before writing it up: it is not. buildLiveGraphIr never derives edges from attributes on any path, so this predates both fixes.

[lex00]

Added chant's board figures on the fixed build, as three replicates rather than one.

A single new run would have joined a replicate set whose other two members are the vendored 0.33.1 — two workspace fingerprints and two harness commits inside one set that the board takes a median across. The figure would have described neither build. The set is homogeneous now:

run	score	workspace	harness
chant-r3	24/24	3912387ce569	4f3ab36-dirty
chant-r4	22/24	3912387ce569	4f3ab36-dirty
chant-r5	22/24	3912387ce569	4f3ab36-dirty

Median 0.9167 — exactly the g-series median. The two chant fixes do not move the board, and that is the expected result rather than a letdown: 0.33.1 read the estate by shelling out to the CLI, which passes --region and signs it, so the arm already saw all three regions. INTENTIUS/chant#1429 is a regression against a build the board never ran.

Where the fixes do move the number is the negative set, which asks the question the board does not: 0/6 → 3/6.

All three runs answered with zero account reads, entirely from the recorded snapshot, and all three passed the audit at 3%, 5% and 1% invocation failure rates.

70/70 records satisfy the contract; mkdocs build --strict clean.

[lex00]

5/6. The subnet question went 1/3 → 3/3; the VPC question is unchanged at 2/3, which was predicted rather than hoped for.

Two more chant defects, both pre-existing, both found by measuring rather than reading:

• fix(aws): a subnet's occupants are traversable chant#1437 — NetworkInterface -> Subnet was declared as containment with no traversal name, so reconstructEdges recorded a boundary hint and emitted no edge. The idiomatic query matched 23 of 23 subnets.
• fix(core): a resource recorded twice replays as one node chant#1440 — a subnet its own stack declares is also recorded as a dependency, so 13 subnets replayed as 16. A duplicate always reads as empty, because the edges resolve to the other copy — the three false positives were the three occupied subnets.

Each fix was necessary; none was sufficient

Measured against the deployed estate at each stage, with ground truth 8 of 13:

build	returns	why
as published (0.33.1)	5 total, 2 empty	one region visible
+ #1429 region	7	regions merged by id
+ #1425 dedup	11	negation inert
+ #1437 traversal	11	duplicates read as empty
+ #1440 duplicates	8 ✅	correct, keeping occupied default-c out

Why the VPC question stays at 2/3

Containment into a VPC is deliberately not traversable — reference-catalog.test.ts asserts expect(edges.some((e) => e.to === "vpc")).toBe(false). My first patch to #1437 flipped that and the test caught it. Flipping an explicit invariant to improve a benchmark number is a decision for its own change, not a side effect of this one, so I reverted it and left the question open on #1437.

The page now lists every run

A second run of an arm is usually a different build, and showing only the newest quietly drops the evidence that a number moved. Both runs are listed with their workspace fingerprints, which is what says they are not the same experiment.

Gates on chant-neg-2: 6/6 trials on chant's own tooling, 41 invocations, 0% failure, 0 account reads. 71/71 records satisfy the contract; strict build clean.