v0.1.6: run full existing benchmarks excluding #122 and report actual numbers

[yifanxuaaa]

Objective

After the Snapshot-Isolated Workspace implementation has a correct, sealed candidate, run the full existing active benchmark suite excluding the benchmark work owned by #122, then publish the actual numbers and verification/resource/cleanup outcomes honestly here.

This issue owns the final phase of docs/roadmap/0.1/0.1.6/overlay-snapshot-implementation-plan.md. It is an execution/reporting issue, not authorization to implement or run #122, invent new performance gates, or replace the existing benchmark framework.

Implementation dependency: #124. Start after its correct sealed-candidate handoff; do not require the implementation issue to be closed first.

Documents and initial status

Use overlay-snapshot-rule.md, overlay-snapshot-spec.md, overlay-snapshot-spec-review.md, overlay-snapshot-implementation-plan.md, and benchmark-exclusions-issue122.json in docs/roadmap/0.1/0.1.6/.

At issue creation these documents are local uncommitted drafts, not published source or measured results. The implementation issue must freeze/publish them and link immutable source before execution. Do not run against a known incomplete snapshot model or fabricate successful qualification from this issue's existence.

Exact scope: full suite minus #122

1. Enumerate the complete active registry and all active family-specific entrypoints from the sealed candidate, not only smoke/default cases, HOST_FAMILIES, or the affected subset. Include inherited required extended/proof-only modes and their existing seed/repetition/source-arm contracts. Keep active unavailable cases in the ledger as blocked.
2. Re-read v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 and reconcile the machine-readable exclusion manifest with its cases.json source/hash and the actual registry. The exclusion snapshot below contains 36 cases: 33 regular and 3 extended. Account for renamed/versioned successors explicitly before execution.
3. Freeze included = active_suite - issue122_owned_cases, along with a disjoint explicit exclusion ledger. Existing older cases in shared families remain included. Do not exclude all dedup_branch_history, mixed_load_bearing or historical_access merely because v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 adds cases there.
4. Exclusion applies to performance, verification, extended variants and case-specific setup. Do not use shared/v016_matrix.py as this campaign: that script targets the excluded v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 matrix.
5. Verify manifest uniqueness, nonempty inclusion, modes and expected cardinality before launch. Report known exclusions separately from exclusions actually registered; planned missing cases do not become invented registry rows.

Execution contract

• Follow benchmark/AGENTS.md, docs/general/benchmark_rules.md, and benchmark/fs-bench-pro/QUICKSTART.md; use existing family/generic runners and the measurement lock.
• macOS host owns Store/SDK/coordinator/canonical construction/spool; Docker owns only Linux daemon/FUSE/workload. Preserve approved limits and source/image custody. No competing builds/perf/verifiers contaminate measurements.
• Reuse compatible sealed binaries/images and protected pristine inputs. Each mutation invocation gets an independent writable copy. No cached post-edit state, post-Commit state or live Workspace as a shortcut.
• Run every included required performance and separate verification row under its existing fixture, API, timing, sample/seed, deadline and verification contract. Proof-only performance is N/A, not zero.
• Compare matched baseline/candidate data only where semantics and custody support it. Otherwise report candidate numbers and baseline UNAVAILABLE/INCOMPARABLE; never invent speedups.
• Introduce no new numeric performance gate. Preserve existing statuses/criteria and all valid failures/misses. Do not shorten work, selectively rerun unchanged code, raise a timeout after a miss or weaken verification.
• If a real defect is fixed, preserve original evidence, reseal the candidate and rerun required affected evidence. Unrelated independent cases should not silently disappear because one case failed.

Required honest report

Publish machine-readable complete matrix and readable tables, with append-only raw manifests and reproduction commands. Each included case/version/seed/arm/mode needs:

• Actual raw timing units and samples, sample count, supported median/range/percentiles, operation/invocation wall under the existing boundary, and useful Init/edit/read/Commit phase timings.
• Host and container CPU/memory/cgroup scopes separately; Store/spool/metadata/index/retained/cleanup disk and I/O where measurable, with missing values marked unavailable and sampling caveats.
• Verification coverage and outcomes; Created/UpToDate/conflict/error; performance, correctness, resources, cleanup and custody statuses kept separate.
• Comparable before/after values and differences, or explicit unpaired/incomparable status.
• Exact product/harness/helper/image/fixture/oracle identities, artifact locations and manifests.
• Expected versus observed cardinality, every missing/blocked row, and a separate EXCLUDED_ISSUE_122 ledger. Excluded rows are not PASS or included NOT_RUN results.

Do not turn missing metrics into zero, reuse archival results as current results, hide failures behind aggregate averages, or claim a fast inner Commit means a passing complete invocation. Publish numbers even when they are worse than expected.

Completion checklist

• Link the correct implementation candidate, immutable documents and source/binary/image identities.
• Publish full active-suite include/exclude manifest and prove v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 is excluded at case level.
• Execute all included required performance rows, with retained samples and exact contract status.
• Execute all included applicable independent verification/proof rows.
• Account for all failures/timeouts/blocked/missing cases and resource/cleanup/custody results.
• Publish full machine-readable matrix, readable actual-number tables, raw manifests and commands here.
• Link benchmark-discovered product defects/fixes and final-source revalidation where required.
• State explicitly that v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 was excluded and that no new performance gates were introduced.

Owner completion requirement supersedes the earlier reporting-only closure allowance: do not close #124 or #125 with required unresolved FAIL/TIMEOUT/BLOCKED/NOT_READY/NOT_RUN, invalid verification, resource, cleanup or custody outcomes. Repair required failures and meet existing applicable mandatory criteria; reporting-only targets remain reporting-only and receive no new numerical gate. Preserve every original failed attempt. Required unexecuted rows leave full-suite execution incomplete. No release, tag, website publication, or changes to #122's evidence are requested.

#122 exclusion snapshot

Source: docs/roadmap/0.1/0.1.6/cases.json; exact machine-readable copy in benchmark-exclusions-issue122.json. Exclude every case below in all modes; do not blanket-exclude their families.

Family	Lane	Excluded case ID
dedup_branch_history	regular	v016-history-large-hotset-k10-v1
dedup_branch_history	regular	v016-history-large-hotset-k100-v1
dedup_branch_history	regular	v016-history-namespace-inode-k10-v1
dedup_branch_history	regular	v016-history-namespace-inode-k100-v1
dedup_branch_history	regular	v016-history-boundary-cycle-k10-v1
dedup_branch_history	regular	v016-history-boundary-cycle-k100-v1
file_size_transition	regular	v016-boundary-small-control-v1
file_size_transition	regular	v016-boundary-below-v1
file_size_transition	regular	v016-boundary-exact-v1
file_size_transition	regular	v016-boundary-above-v1
file_size_transition	regular	v016-boundary-large-control-v1
file_size_transition	regular	v016-boundary-roundtrip-v1
file_size_transition	regular	v016-boundary-alias-roundtrip-v1
mixed_load_bearing	regular	v016-mixed-development-100mb-5000-k10-v1
mixed_load_bearing	regular	v016-mixed-development-100mb-5000-k100-v1
mixed_load_bearing	regular	v016-mixed-development-500mb-30000-k10-v1
mixed_load_bearing	regular	v016-mixed-development-500mb-30000-k100-v1
multi_workspace_development	regular	v016-workspace-mixed-100mb-5000-k10-v1
multi_workspace_development	regular	v016-workspace-mixed-100mb-5000-k100-v1
multi_workspace_development	regular	v016-workspace-mixed-500mb-30000-k10-v1
multi_workspace_development	regular	v016-workspace-mixed-500mb-30000-k100-v1
branch_development	regular	v016-branch-mixed-100mb-5000-k10-v1
branch_development	regular	v016-branch-mixed-100mb-5000-k100-v1
branch_development	regular	v016-branch-mixed-500mb-30000-k10-v1
branch_development	regular	v016-branch-mixed-500mb-30000-k100-v1
branch_development	regular	v016-branch-convergent-content-v1
branch_development	regular	v016-branch-fork-descendant-v1
historical_access	regular	v016-access-boundary-before-v1
historical_access	regular	v016-access-boundary-after-v1
historical_access	regular	v016-access-inode-before-v1
historical_access	regular	v016-access-inode-after-v1
historical_access	regular	v016-access-fork-point-v1
historical_access	regular	v016-access-divergent-head-v1
mixed_load_bearing	extended	v016-mixed-exhaustive-100mb-5000-k100-v1
mixed_load_bearing	extended	v016-mixed-exhaustive-500mb-30000-k100-v1
multi_workspace_development	extended	v016-workspace-four-100mb-5000-k100-v1

[yifanxuaaa]

Phase 1 mirror from #124: documents frozen, scope ledger recorded, campaign not started

No benchmark case has been executed, prepared, or selected for execution. This comment records the benchmark-facing state so #125's later campaign does not start from an unfrozen catalog, and states why Phase 7 cannot begin yet.

Frozen source of truth (published)

86f6a0a0ff3de2d524b4984ab9c6b5de5272b18b
publishes the rule, specification, adversarial review, architecture, implementation
plan, handoff and benchmark-exclusions-issue122.json.
a7b565640bc58c794af7de552ff71eb416d0bd45
adds the contract resolution (Phase 7 scope method in §6), the progress/resume state and the append-only verification ledger.

Provisional include/exclude ledger (enumeration only)

Emitted by the existing registry entrypoint fs-benchmark-pro infra-list; no case was run and the measurement lock was not taken. Raw output and the machine-readable ledger:
evidence/phase1-catalog/scope-ledger.json.

Item	Count
Active registry scenarios / families	260 / 21 (258 supported, 29 proof-only)
#122 exclusion manifest declared	36 (33 regular, 3 extended)
E ∩ U (declared and currently registered)	28, every one v016-prefixed, none in a family excluded wholesale
E \ U (declared, still unregistered)	8 — six historical_access v016-access-*, two branch_development (v016-branch-convergent-content-v1, v016-branch-fork-descendant-v1)
Included registry scenarios	232
Family-specific entrypoints outside the generic registry	historical_access 11 cases (historical-access-v2), repository_history 3 optional profiles (NOT_RUN_OPTIONAL unless selected), small_file_delta_smoke 1 exploratory case
Provisional counted inclusion	243 rows (232 + 11)

Every #122 row is EXCLUDED_ISSUE_122 in the ledger, not PASS and not NOT_RUN. shared/v016_matrix.py is not the Phase 7 driver: it targets the excluded matrix. Claims in docs/roadmap/0.1/0.1.6/cases.json (status: planned-unimplemented-unqualified, implemented: false for all 36) disagree with the live registry for the 28 registered IDs; the registry is authoritative for execution and the discrepancy is recorded rather than resolved by inventing registry rows.

Why Phase 7 has not started

Phase 7 requires a correct sealed candidate from #124. Phase 1 of that work ended by proving a platform incompatibility instead of an implementation: a store into a writable MAP_SHARED mapping of a cached FUSE file is not visible to the daemon until an explicit writeback trigger, mmap sends the daemon no request at all, and the only triggers are per-file fsync/msync/munmap (unavailable for open-unlinked inodes) or mount-wide syncfs (work proportional to dirty payload). Exact mapping visibility, non-freezing Commit and bounded acquisition therefore cannot all hold under the current rules, and the choice belongs to the owner (options in the resolution document, Phase 1 comment on #124). Running the suite now would measure the unchanged frozen product, which this issue explicitly forbids ("do not run against a known incomplete snapshot model").

Unchanged constraints for the eventual campaign

No new numerical performance gate, capture target or replacement family is introduced. Proof-only rows report performance N/A, never zero. Missing metrics are unavailable, never zero. The campaign reuses sealed binaries/images and protected inputs under the existing measurement lock (macOS host owns Store/SDK/coordinator/canonical construction/spool; Linux Docker owns daemon/FUSE/workload), and every #122 case stays excluded in all modes and case-specific preparation.

[yifanxuaaa]

Execution takeover and immutable source freeze

Implementation continues under #124 takeover. The scoped local handoff/plan/progress edits were preserved and frozen at 554b866838e50468b312f9d8e9ef70dc2988a38e. Authoritative rule, spec, plan, and 36-case exclusion manifest are immutable source.

V1 remains open; a new retrieval investigation corrects the predecessor's no-retrieval assertion but refutes retained page references as snapshots. Dependency-ready implementation proceeds. No correct candidate is sealed yet, no benchmark row has executed, and no #122 case/setup is authorized or run. Phase 7 starts from a correct sealed candidate without waiting for #124 closure. The owner's stronger terminal requirement governs closure: required failures must be resolved; reporting-only targets stay reporting-only and no new numerical gate is introduced.

[yifanxuaaa]

Benchmark-facing implementation checkpoint; no Phase 6 handoff yet

Source 3f4a26d4a adds host overlay resource admission and physical/logical accounting, numeric directory cookies, host operation/replay and owned reader integration. Component raw evidence and source identities are in the verification ledger and host-operation evidence. These are development correctness checks; no timing is presented as a benchmark result.

The candidate is not sealed: generic-FUSE V1 mapping isolation and production client/canonical builder/lifecycle integration remain open. Phase 7 has not started, and no #122 benchmark was executed. The exact #122 exclusion manifest remains retained; final execution will reconcile the full active registry/family-specific entrypoints and exclude all 36 recorded #122 identities/successors before launch. #125 stays OPEN and will require terminal success under its existing applicable contracts.

[yifanxuaaa]

Implementation checkpoint published; benchmark campaign remains NOT_STARTED

The owner's bounded stopping instruction was fulfilled for the three integration fixes: mounted shutdown/unmount, real-spill capacity fixture and retained-stage maintenance oracle all have focused passing evidence, with original failures preserved. Main merge 74b2bdb93, PR#126, successor handoff, checkpoint result matrix/raw logs.

This is not the Phase6 sealed-candidate handoff or Phase7 completion. Generic V1 and full public-path/capacity/lifecycle acceptance remain open. The mounted component uses explicitly supplied owned input; no full benchmark row is reported PASS. No final inclusion/mode/seed/repetition manifest or performance matrix has been produced.

CI alone now has the owner-authorized180second budget. The171s native run's original150s gate failure is retained, as is the historical125s/120s failure. No benchmark deadline or numerical criterion changed. Strict Clippy remains unresolved in the handoff.

The current provisional catalog/exclusion identities are retained: scope ledger, exact36-case exclusion manifest,33 regular+3 extended. Reconcile from the eventual correct sealed candidate before launch, including family-specific entrypoints. Exclusions apply to all modes and case-specific setup, not entire shared families. No#122 scenario was executed, no final benchmark campaign ran, and both#124/#125 remainOPEN. No release/tag.

[yifanxuaaa]

Benchmark-facing status from the #124 capacity work (no benchmark has been executed).

Two shared-source capacity repairs are now merged to remote main, and they change the resource envelope a future #125 campaign would measure rather than any benchmark definition:

• Remote main is at ea34ba98a521c63d305010118af4f14e30c95683 (PR #128, PR #129).
• Payload ownership no longer conflates resident handles with persisted tokens, and the payload catalog now reclaims retired pages, so a Workspace's changed-file capacity is bounded by its explicit index/arena disk quotas instead of an accidental 8,192 resident-owner ceiling.
• The ordinary SnapshotReader demand cache (8 MiB) is now charged to the Workspace's fixed resident reservation instead of being admitted for free.
• Canonical scratch, including admission-owned spill, is created in the owning Workspace directory rather than the shared process temporary directory, and full-attempt scratch peaks are recorded separately from construction-only peaks.

No benchmark case was run, no #122-owned scenario was executed, and no case identity, mode, repetition, gate or timeout changed. The #122 exclusion set remains 36 exact cases (33 regular, 3 extended); excluded rows stay EXCLUDED_ISSUE_122. The Phase 7 campaign still requires a sealed candidate, which in turn requires the open V1 and public-path exits. Both issues remain OPEN.

[yifanxuaaa]

Benchmark-facing status: no benchmark has been executed, and the Step 2-3 public-path migration changes the measured surface rather than any benchmark definition.

• PR #131 makes the ordinary public Workspace route publish through one host authority with an owned snapshot cut and no freeze. For a future v0.1.6: run full existing benchmarks excluding #122 and report actual numbers #125 campaign this changes what the harness exercises on the default path; it changes no case identity, mode, seed, repetition, gate, timeout or exclusion.
• PR #132 is documentation only and ships the V1 investigation evidence.
• No performance number is claimed by either. The held-construction wall times in the tests are correctness context, not latency measurements.

The #122 exclusion set remains 36 exact cases (33 regular, 3 extended); excluded rows stay EXCLUDED_ISSUE_122 and are never recorded as PASS or NOT_RUN. Phase 7 still requires a sealed candidate, which requires the open V1 decision and the remaining Step 1/5 exits. Both issues remain OPEN.

[yifanxuaaa]

Benchmark handoff updated: five tier-500 cases first; scale cases deferred

Latest owner direction supersedes the earlier 25k/two-second milestone: quick iteration targets the five existing tier-500 cases below, seeking close performance under the existing paired regression screen and stronger case criteria. The 25,000-file and million-file cases are deferred, not passed or waived. Do not register or run them for this evaluation.

• tiny-create-500-mixed-v4
• tiny-stat-500-mixed-v4
• tiny-unlink-500-mixed-v4
• tiny-bulk-create-500-mixed-v3
• tiny-bulk-delete-500-mixed-v3

Bulk tier 500 means 5,000 files / 500 MiB; the other three cases affect 500 files. Preserve fixtures, normalization/root sync, timers, seeds and independent proof. Establish results for the five, then rerun only failing or genuinely affected checks. Full 20-case tiny-churn qualification follows when stable, not after every fix.

Current immutable implementation plan · Review PR #135

Keep each inherited pure_call_sum and existing applicable criteria, including stronger registered absolute targets. The existing paired regression screen applies to a complete selected workflow (receipt operations=1); do not divide its delta by file count to evade the floor. Historical v0.1.5 values remain context until matched source/fixture/harness/topology/cache comparisons are obtained.

All 20 inherited tiny-churn identities are outside the 36 #122 exclusions. The later full #125 campaign retains every included regular/required extended case under its existing contracts; a development quick screen is not final campaign evidence. Deferred capacity is not a PASS or waiver of this issue's terminal prerequisites. No benchmark, product test or probe ran for this documentation update; #124 closure is not a handoff prerequisite.

[yifanxuaaa]

#125 benchmark-facing note from #130 (P130.2): five/20-case comparison BLOCKED on route, counters available

The current #130 quick-iteration five-case screen and the later 20-case tiny-churn family were not run, and no #122-owned case or shared/v016_matrix.py campaign was touched. Reason, with source evidence: in the benchmark topology (macOS host store/coordinator + Linux Docker workload) container placement mounts through projection.rs WorkspacePlacement::Container → live_backing::RemoteWorkspace + docker::DockerProjection::attach (legacy remote protocol); the new host authority is installed only on the Host + Projection::Fuse path. A legacy-route candidate run cannot qualify the #130 implementation, so the comparison is reported BLOCKED rather than substituted. Full reference: #130 (comment) , mirrored to #124.

What #130 does have for #125's future campaign (component-level, not case evidence and not a substitute for it):

• Measured per-tiny-file counters through the ordinary host-authority create/write path: create 62.9 metadata page writes / 378 page reads, first write 21.7 / 126, payload catalog 8 allocated pages, payload 4 KiB physical per file (host_overlay::tests::tiny_create_metadata_page_cost_is_measured).
• Byte/scaling oracles for bounded reclamation: repeated release/replace churn at 16/32/64 live 8-KiB files relocates exactly the freed bytes (131,072 / 262,144 / 524,288) with 0 unclaimed bytes; the pre-repair source relocated 33,030,144 B for 524,288 B freed.
• Whole-suite gate: tools/test-fast.sh PASS, 157s with 4 bounded jobs, Rust 1.85.1.

#125's own case list, exclusions (36 #122 cases), custody and terminal requirements are unchanged; the deferred 25k/two-second and million-file cases remain DEFERRED/OPEN and were not registered or run.

[yifanxuaaa]

#125 benchmark-facing: #130 M2 first paired screen on the wired route (two cases complete, ~80× absolute gap)

Recorded for #125's later full campaign — this is not campaign evidence and does not complete any #125 requirement. Owner stopped the run after the first results.

• Topology: macOS host store/coordinator + Linux container (2 CPU / 2 GiB / no swap / 256 PID, /dev/fuse, SYS_ADMIN), container daemon mount route, perf-fast collection mode, seed 1, --setup clone, n3 fresh alternating pairs.
• Arms: candidate = main 918ad73de (binary 5dc3fd49…, image layerfs-bench-infra:95aa0eada65960a8); control = same wired route with the three Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130 commits reverted (codex/issue130-m2-control 0b15e1783, binary 2f258032…, image layerfs-bench-infra:9b0bdbcb2507eb12). Custody verified from receipts.
• Results: tiny-create-500-mixed-v4 median 20,029.7 ms vs 19,395.8 ms (+3.27 %, 2/3 pairs slower, threshold 2,909.4 ms) → PASS, no material regression; tiny-unlink-500-mixed-v4 9,323.4 ms vs 9,423.6 ms (−1.06 %, threshold 1,413.5 ms) → PASS.
• Absolute: ≈80× the historical v0.1.5 create-500 sum (242.659707 ms); phases exec ≈11.4 s, commit ≈4.4 s, end ≈4.2 s, visibility µs.
• Incomplete/not run: tiny-stat-500-mixed-v4 one usable pair (+8.64 %), tiny-bulk-create-500-mixed-v3 and tiny-bulk-delete-500-mixed-v3 not run; no counter-level dispatch proof in the receipt; no v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 case touched; shared/v016_matrix.py not run.
• Evidence: PR docs: record the #130 M2 paired screen results and the measured slowness #141 (merged 8dc7effad), ledger L44; raw receipts in the git-ignored benchmark-results/issue130-m2/runs/. Full detail: Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130 (comment)

[yifanxuaaa]

#125 benchmark-facing mirror from #144 Phase 1 (PR #146, ledger L46): the run receipt schema gains counter fields (host_authority_* dispatch counts in FuseWriteReceipt; commit pre_capture_ns/maintain_ns/construction_tail_ns/stage_ns/acknowledge_ns + construction_* diagnostics; a new HostEndReceipt with settle/after_detach/state-removal splits) — schema-additive, no benchmark-harness change required. One measurement note for the later campaign: tiny-bulk-create-100-mixed-v3 currently FAILs to complete (scratch allocation exceeds admitted growth after 28.8 s exec) — not campaign evidence, but the campaign cannot include that case until the admission issue is resolved. Single-sample Phase-1 diagnostics are not campaign evidence; the M2 n3 medians remain the ledger baseline.

[yifanxuaaa]

Superseded as an overlay-plan execution issue — closing as not planned, and explicitly not claiming the campaign ran.

The full-suite-minus-#122 campaign this issue owns was never executed, under either model. The adopted sandbox-local model has its own sequential gates in #151 (B1 FAIL at the complete-Commit phase, B2 memory metric unresolved, B3 unrun) and a full release-style campaign that is still to be scheduled and currently owned by nobody. Its exclusion manifest benchmark-exclusions-issue122.json is absent from main (archive tag archive/main-branch-cleanup-20260915-1005/heads/archive/v016-overlay-7b73c4b33).

If the owner wants the full-suite obligation tracked, it should be opened fresh against the sandbox-local route with the current registry rather than resumed here.