v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit

[yifanxuaaa]

Objective

Implement v0.1.6 Snapshot-Isolated Workspace: one current mutable overlay with shared host backing, an owned temporary snapshot per Commit attempt, and independent canonical construction -> workspace_stages -> conditional branch publication. Reads, writes, namespace changes, SDK operations, running commands, mount identity and open handles must remain valid while Commit runs.

This is product implementation work, broader than the pending-metadata capacity objective in #123. Coordinate with #123 without silently replacing or closing it. Benchmark issue #125 owns the implementation plan's final full-suite execution/reporting phase. #122's benchmark campaign is explicitly outside this work.

Source of truth

The owner-discussed documents are under docs/roadmap/0.1/0.1.6/:

• overlay-snapshot-rule.md
• overlay-snapshot-architecture-design.md
• overlay-snapshot-spec.md
• overlay-snapshot-spec-review.md
• overlay-snapshot-implementation-plan.md
• benchmark-exclusions-issue122.json

At issue creation these are local uncommitted drafts in the main working checkout. They are not published immutable source, implemented behavior, or benchmark results. Freeze/commit/push the authoritative documents at the start of implementation and attach immutable links here. This issue body is self-contained scope until those links exist.

Required resulting behavior

• Commit does not freeze/quiesce the Workspace for construction/publication, require writers/commands to finish, remount, or install an older captured state back into live inodes.
• Ordinary mutations atomically update current state and change tracking. There is no per-file checkpoint, operation-history log, or automatic intermediate branch Commit.
• Snapshot acquisition is bounded root/ownership acquisition; no changed-map clone, full dirty-fact export, complete cache drain, or delayed whole-overlay clone on the first later write.
• Preserve one consistent captured view of namespace, attributes and content, including the actual supported FUSE cache/mapping contract. Do not pretend a host root alone captures kernel-visible bytes.
• One unresolved Commit attempt per Workspace; no global construction serialization across independent Workspaces. Existing branch leases and expected-head/base publication checks remain.
• C1 excludes a change x installed after its snapshot; C2 includes x if still effective at its own snapshot. Publication advances only exact covered/comparison context and leaves live contents untouched.
• Keep the existing shared Init/Commit content/output/admission machinery, CAS/authentication, FULL/DELTA policies, large-file CDC/extents, packing/compression, sorted namespace updates and staged publication.
• Make indexes/caches/ownership/cursors/queues/FDs and transient allocation bounded. Reclaim superseded state by ownership; preserve held readers and open-unlinked lifetime.

Resolve known specification blockers honestly

V1 kernel visibility, V2 page/catalog/ownership/backend and retry-fairness details, V3 bounded predecessor correspondence, and V4 exact uncertain Created/UpToDate resolution must be made concrete and proved. If a compliant kernel mechanism cannot be established, record the precise blocker; do not silently disable mmap, require implicit user fsync, weaken visibility, or restore a global freeze. V5 is later evaluation using existing benchmarks, not a new numerical performance gate.

Removal / addition boundaries

Add/adapt: host snapshot-capable backing/root bundle, bounded snapshot reader/cursors, independent attempt context, atomic dual change indexes, metadata-only canonical/range correspondence, bounded replay and reclamation.

Remove from ordinary Commit: pause/quiesce/resume and writer-finish dependency, long live/fact locks, mandatory checkpoint reinstallation, global dirty/generation reset, pending-stage/publication-as-inactive, and retries rebuilt from newer current state.

Replace: full dirty-prefix export/mirrored maps as Commit input, unbounded piece/map/path access, whole-extent-only retention and unbounded segment-descriptor growth where the new model requires it.

Preserve/audit: shared canonical engine, workspace_stages, conditional publication and all independent fsync/cache/SDK/reconciliation/End/Discard consumers. Do not delete shared helpers solely because Commit stops using them.

Plan and acceptance

• Resolve and freeze the actual implementation contracts and attach source links. Governing documents and V2/V3/V4 decisions are published; V1 remains OPEN. The predecessor's Phase1-complete/universal-incompatibility/owner-wait classification is superseded by the retrieval correction: checked generic FUSE retrieval retains mutable pages. No semantic relaxation is approved; dependency-ready implementation continues.
• Implement host overlay ownership and atomic current state with source-root lease/revalidation; unrelated writers cannot lose updates.
• Integrate FUSE/SDK operation semantics and owned snapshot access without global Commit exclusion.
• Integrate independent attempts, unchanged shared construction, exact staging/publication and safe C1/C2 correspondence.
• Remove obsolete ordinary-Commit coupling and finish failure-safe cleanup/explicit lifecycle handling.
• Pass applicable existing correctness/integration checks plus focused missing coverage: root races, replay, tombstones, zero-range alignment, unequal edits, kernel visibility, retained readers, open-unlinked files, conflicts/uncertainty and cleanup.
• Complete v0.1.6: spill pending Workspace metadata to disk for bounded-memory large change sets #123's explicit million-changed-regular-file correctness/capacity proof in one Workspace before one final Commit, with declared RAM/disk/work/deadline and fresh reopen; no arithmetic-only or incremental-Commit substitute.
• Seal a correct candidate and hand it to the benchmark issue v0.1.6: run full existing benchmarks excluding #122 and report actual numbers #125 for the plan's last phase: full existing benchmark suite excluding v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122.
• Link final benchmark evidence and track any discovered product defects; do not claim implementation correct with unresolved correctness failures.

Performance policy and exclusions

Implement correctly first, then run the existing sophisticated benchmark suite and inspect/report real numbers. No new capture-latency, throughput, or sampling gates. Existing benchmark contracts remain unchanged.

The final benchmark campaign excludes all 36 #122-owned cases in cases.json (33 regular + 3 extended), in every mode and case-specific preparation. Exclude exact cases, not entire shared families: inherited cases in dedup_branch_history, mixed_load_bearing, and historical_access remain eligible. Reconcile renamed/versioned successors with #122 before freezing the suite manifest.

No implementation or benchmark was executed merely by opening this issue. No release/tag/website publication or completion of #122 is requested.

Benchmark execution/reporting: #125. Handoff requires a correct sealed candidate, not closure of this issue; final campaign evidence is linked back here.

[yifanxuaaa]

Phase 1 complete: resolve contracts and freeze the source of truth

Short version: Phase 1's exit is met through the plan's documented-incompatibility path. V2/V3/V4 are resolved to implementable contracts. V1 (kernel visibility of writable mappings) is now a proved platform incompatibility that needs an owner decision, so #124 is not terminally successful and stays open. No product source was changed and no benchmark case was executed.

Changes

• V1 established empirically, not by argument. A dependency-free probe serves a cached write-through FUSE session (FUSE_ASYNC_READ|FUSE_BIG_WRITES, i.e. no FUSE_WRITEBACK_CACHE, matching crates/layerfs-fuse/src/filesystem.rs:37-47), maps a file MAP_SHARED/PROT_WRITE, stores into it and reports what the daemon saw: mapped stores are invisible (0 WRITE requests, 1000 ms); syncfs on the mount delivers them; a per-file daemon self-open + fsync also delivers them without any mount barrier; that per-file path fails for an open-unlinked inode (ENOENT), which only mount-wide syncfs covers. Kernel source confirms mmap sends no request at all and FUSE_DIRECT_IO_ALLOW_MMAP switches the inode into caching I/O. Mechanisms M1–M6 (mount syncfs, per-file flush, removing writable mappings, host-only snapshot, real-file/passthrough backing, reading pages out of the writer's address space) are each rejected against the rule's non-freezing, no-drain, bounded-acquisition and no-weakening clauses; the decision required is stated as three explicit options.
• V2 resolved: Arc<OverlayRoot> as the single installed authority with structurally shared typed indexes, leased-source-root installation (whole-root identity compared, prepared sequence/index keys, no live lock across I/O), dual change_by_key/change_by_seq tracking replacing LiveWorkspace.dirty/mutation_paths as Commit input, O(1) acquisition over the existing one-word BackingRef ownership, arena-backed payload with declared per-backend block release (PUNCH_HOLE/F_PUNCHHOLE), deferred charged reclamation, bounded (session, request_sequence) replay, and explicit ResourcePolicy caps.
• V3 resolved: metadata-only provenance descriptors per published inode plus the (origin_id, origin_start) disjoint-interval index and the reviewed two-pass nonzero-anchor/zero-gap matching, feeding the existing ObjectBuffer::set_physical_predecessor/PredecessorCursor path. This removes the concrete FrozenFile::file_may_differ → file_matches full-file comparison at crates/layerfs-workspace/src/changes.rs:1843-1854 that would otherwise make every small C2 a full rebuild.
• V4 resolved: one minimal explicit Store receipt table written inside the existing publication transaction, plus an exact four-case resolution procedure that never treats a missing stage as evidence and uses the deterministic CommitId::derive(root, parent, base) and the conditional ADVANCE_BRANCH as an independent witness for Committed.
• Removal/audit inventory recorded against current call sites (pause/quiesce gates, fact export, checkpoint installation, ensure_active coupling, flush_kernel_cache, and the must-audit non-Commit consumers), plus Phase 7's provisional scope ledger (232 included registry scenarios, E ∩ U = 28, 8 exclusions still unregistered, 11 historical_access cases).

Source

• Contract resolution, progress/resume state and append-only ledger: overlay-snapshot-contract-resolution.md, overlay-snapshot-progress.md, overlay-snapshot-verification-ledger.md.
• Frozen authoritative documents published in 86f6a0a0ff3de2d524b4984ab9c6b5de5272b18b (rule, spec, spec review, architecture, plan, handoff, benchmark-exclusions-issue122.json); Phase 1 commit a7b565640bc58c794af7de552ff71eb416d0bd45 on main. Source baseline named by the spec remains 0814cc37f1dafb6041930c74489107f4a5035a26.
• No product binary/image is claimed: no product code was written, so no candidate exists yet.

Verification

• V1 probe: evidence/v1-probe/README.md with probe.c (sha256 bd38ad80…8d608a), run.log (sha256 00d1eb9e…1cdca3) and SHA256SUMS; image rust:1.85.1-bookworm, kernel 6.12.76-linuxkit, /dev/fuse + SYS_ADMIN. Ledger entry L1.
• Benchmark catalog enumeration (fs-benchmark-pro infra-list, registry only, nothing executed): evidence/phase1-catalog/scope-ledger.json and infra-list.jsonl; 260 scenarios / 21 families / 258 supported / 29 proof-only. Ledger entry L2, explicitly provisional for Phase 7.
• Ledger entry L3: git diff over crates/ and benchmark/ against 86f6a0a0f is empty, so every previously recorded product pass stays valid for the unchanged product.

Reused passes

• All existing v0.1.6 (v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122-side) and v0.1.5/v0.1.5 finalization: full benchmark campaign, honest dispositions, no measurement cheating #120 rows are retained unchanged, justified by L3 (byte-identical product source, unchanged fixtures/oracles/images). None of them qualifies the new design: they describe the product whose ordinary Commit still freezes and reinstalls a checkpoint, which is exactly what this work must remove.

Failures resolved

• None yet: no failing product test was fixed in this phase, because no product change was made.

Remaining

• Owner decision required (V1). One of: (1) permit a bounded acquisition-time writeback barrier (per-file flush plus mount syncfs for open-unlinked inodes) with the rule's bounded-acquisition clause narrowed to "bounded by kernel-dirty state"; (2) narrow the supported surface by refusing writable MAP_SHARED on cached handles, re-versioning the existing tests that assert it; or (3) keep the current freeze+syncfs behavior and stop the feature. Everything else in Phases 2–5 is identical under (1) and (2) and can start as soon as one is chosen; under (3) the work stops.
• Phases 2–6 not started; Phase 7 not started and must not run against an incomplete snapshot model. No benchmark row has been produced, no new numerical gate was introduced, and no v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122-owned case was executed or prepared.

[yifanxuaaa]

Phase 1 correction (same phase, no status change): two contract defects found and fixed

An adversarial pass over the Phase 1 contracts I published in a7b565640 found two defects before any implementation depends on them. Both are fixed in 56c8d29c6; Phase 1's status is unchanged (V2/V3/V4 resolved, V1 awaiting the owner decision, #124 open).

1. V4 receipt key was in-memory. Keying the workspace_publications receipt by an attempt identity cannot resolve uncertainty after a process restart, yet v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124 requires retrying the same candidate and expected context rather than rebuilding from newer live state. The key is now the deterministic tuple (workspace_id, candidate_root, expected_head, new_base_layer), which a retry recomputes and which also reproduces CommitId::derive(root, parent, base), so the Committed case keeps an independent witness.
2. V2 lacked the substitution obligation. Without it, live ranges that still reference temporary bytes which have already become canonical would pin the temporary spool indefinitely, which rule §5 forbids. §2.10 now specifies per-range exact-equivalence backing substitution that retains canonical ownership, changes no content/offset/identity/generation, installs no captured state, and charges bytes that cannot be safely substituted yet.
3. V1 wording tightened. Neither mount-wide syncfs nor a per-file fsync yields an observable cut: a store that completes before the flush returns can still miss the kernel's internal collection point. Option 1 in the resolution is therefore stated as "boundary = the kernel's collection point inside the flush", not as a clean per-file boundary. This refines V1; it does not weaken the blocker or resolve it.

Ledger entries L4 and L5 record both. Still no product source change, still no benchmark execution, and the required owner decision in §1 is unchanged.

[yifanxuaaa]

Phase 5 prerequisite recorded: non-Commit consumer audit (f996a56d7)

Before any Phase 4/5 deletion, the helpers the new design removes were audited against their actual callers (resolution §5.1, ledger entry L6). Three findings change the removal plan:

1. The cut gate is not Commit-only. The ordinary SDK splice path takes the same cut with publish = false before reading live-owner state (live_owner.rs:2956-2961), prepare_shutdown takes it (:2568), and gate.cache_flush() is used directly at :1936. Commit must therefore stop using the complete-dirty-prefix flavour rather than the primitive being deleted; deleting it would break ordinary edits and shutdown.
2. The container installs checkpoints independently. live_owner.rs:363-380 (validate_checkpoint_record/install_checkpoint_record/finish_checkpoint, driven by INSTALL_BEGIN/INSTALL_END at :3224-3235) is a separate path from the host live_backing.rs:2080 install that lifecycle.rs:642 calls. Removing the host Commit call does not remove that machinery and must not.
3. syncfs is also the writeback-error channel. live_owner.rs:2619-2621 documents that the call is how the product observes kernel-side writeback errors including allocation failures. A Commit that no longer performs it changes error-reporting semantics, so the replacement contract must state where those errors are observed and delivered — this is now a declared consequence of the V1 decision rather than an implementation detail.

Also recorded: Workspace::ensure_active gates every read/write/namespace operation (cow_tree.rs:224,585-692, execution.rs:175, file_io.rs:409,491,523), and the similarly named ensure_active in objects.rs:2570 is an unrelated admission-session helper that must not be conflated with it.

No product source changed; #124 remains open on the V1 decision.

[yifanxuaaa]

Execution takeover: dependency-ready Phase 2 implementation starts now.

Retained evidence: predecessor commits a7b565640, 56c8d29c6, bfb06a019, f996a56d7, including V1 probe, provisional catalog and L1–L6. The dirty writable-mapping result remains an unresolved acceptance obligation; V2/V3/V4 contracts are not implementation passes.

Frozen source: successor requirements and scheduling correction, 554b866838e50468b312f9d8e9ef70dc2988a38e, retaining the authoritative rule/spec/review/architecture and exclusions from 86f6a0a0f. All three pre-existing scoped local changes were preserved. Implementation branch: codex/snapshot-isolated-workspace.

Next concrete task: implement the host overlay's leased-source-root installation and atomic change indexing, with a focused two-disjoint-writer regression proving stale prepared roots cannot erase unrelated writes and retained roots remain readable. Continue independent ownership/reader/publication work while testing distinct compliant V1 hypotheses; no semantic relaxation is assumed. Full supported-surface correctness, capacity proof and Phase 7 remain pending. No benchmark or #122 scenario has run, neither issue is eligible for closure, and required failures must be resolved before terminal success.

[yifanxuaaa]

V1 evidence correction; implementation continues (no phase completion claim)

The predecessor's statement that Linux provides no kernel-buffer retrieval facility was incorrect. A distinct FUSE_NOTIFY_RETRIEVE probe retrieved dirty mapped bytes, including after unlink by inode ID, with zero WRITE callbacks and no flush. However, page references retained when notification returned did not isolate content: A was present at notification, a subsequent mapped B completed, and the delayed reply contained B. Retrieval supplies visibility but does not supply a consistent retained snapshot. V1 remains OPEN under the unchanged contract.

Source/evidence: 333f3caca56f5d3eb042709217113216da4aa3ef, report and pinned kernel-source analysis, raw probe output, exact command/image receipt. Original probe and all L1–L6 observations are preserved; L7 supersedes only the unsupported exclusivity conclusion. The new probe ran once, under the measurement lock; no benchmark or #122 scenario ran.

Independent Phase 2 code is now in progress: disk COW indexes, leased-source atomic root installation, latest dual change indexes, bounded replay and owned range cursors. Phase 4 Store receipts are progressing independently. A cursor component check has executed one passing test; it is not public snapshot acceptance. Payload ownership, complete FUSE/SDK authority integration, correspondence, cleanup, correctness/capacity, candidate sealing and the final non-#122 campaign remain required. Both issues stay open.

[yifanxuaaa]

Implementation checkpoint: owned host snapshot content and physical ownership (Phases 2–4 still in progress)

Changes: c95588f30 on PR #126 integrates disk COW roots, fixed 128-byte inode metadata, atomic forward/reverse bindings and canonical identity, bounded FIFO mutation preparation, independent owned snapshot reads, disk-linked implicit-offset file ranges, shared payload arenas, and metadata-only predecessor matching. No whole phase is marked complete; public host-operation/FUSE/SDK/candidate/coordinator integration and aggregate resource admission continue next.

Verification: 8 targeted overlay/snapshot checks PASS, including captured captured versus newer live content after dropping the live owner. Index manifests bind 13 compatible checks plus retained admission; range evidence records 4 distinct PASSes; correspondence records 4 component PASSes. Payload evidence measures 67,108,864 -> 4,096 -> 0 actual allocated payload bytes, moving only 4,096 bytes for one retained byte; metadata/catalog allocation is reported separately. These are component measurements, not benchmark numbers.

Failures resolved: original ambiguous-header receipt failure leaked three metadata pages; read-only linked-root traversal exhausted tickets; cleanup status omitted queued work; a retained old reader wrongly blocked independent writes; append through a truncated token consumed input before rejecting it. All original FAILs/compile blocks and exact targeted repairs are preserved in the linked ledgers. Unaffected passes were retained; the unchanged replay test was explicitly skipped in the latest root run.

Remaining: generic-FUSE V1, host-operation/candidate/attempt integration, complete quota/admission and cleanup, public correctness/capacity, candidate seal and final full non-#122 campaign. The owner clarified that the solution must be generic FUSE: LinuxKit is only the probe environment, not a product dependency or an approved custom-kernel requirement. Neither issue is eligible for closure. CI's initial formatting and Store Clippy errors are repaired, but unused-component errors remain until actual production integration; no lint gate is disabled. No benchmark or #122 scenario was executed.

[yifanxuaaa]

Implementation checkpoint: host operation and ownership core (Phases 2–4 remain in progress)

Changes: ordinary host file/namespace operations now use the owned overlay graph, preserving cold canonical lookup, hardlink/open-unlinked identity, atomic rename/splice and stable numeric directory cookies without a daemon-sized directory map. Added aggregate resource admission, separate logical spool/SDK-inline accounting, counted kernel references, explicit backing sync, and an authenticated host-operation codec/handler with bounded replay. Snapshot streaming reads no longer allocate one payload-owner token per read buffer. Store/candidate production integration remains in progress.

Source: commit 3f4a26d4a, draft PR #126. This is a tested component checkpoint, not a sealed benchmark binary/image.

Verification: five host operation/quota/kernel-reference checks, two wire checks, two exact generation/replay component checks, numeric directory-cookie and host-dispatch checks, logical payload accounting and two explicit-sync checks. Nine shared range/description/snapshot-reader prerequisite checks also pass. Exact commands, source hashes, raw logs and statuses are in the host evidence, policy/payload/sync manifests, reader prerequisites and append-only ledger. Directory test covers 301 cold names plus dots across three bounded pages and verifies deleted/churn cookie rows are removed. These are correctness checks, not benchmark numbers.

Reused passes: unchanged core/index/Store cases remain in the ledger. Host checks were rerun only for changed range summaries, snapshot read path, binding/ownership and quota helpers. The directory check was retained when only replay digest semantics changed.

Failures resolved: namespace-root initialization incorrectly used non-root reference validation; a kernel-page test incorrectly equated scanned slots with visible entries; an adversarial retry test proved that a whole-envelope digest incorrectly rejected an unchanged mutation when its cumulative acknowledgment advanced. Each original attempt is preserved with its targeted successful repair. PR CI's original format/Store lint defects are repaired, but latest published CI still fails on unintegrated private components; no lint gate is disabled or CI success claimed.

Remaining/next: compile the actual shared canonical candidate and per-Workspace attempt coordinator; verify held-builder C1/C2, stage/receipt retries and bounded scratch; integrate the generic FUSE client and cache/SDK/lifecycle consumers. V1 remains OPEN. The owner requires generic FUSE: LinuxKit is only the Docker Desktop test-kernel identity, and no custom kernel/VM dependency or semantic relaxation is approved. Independent execution continues. No phase completion, final candidate, benchmark campaign or issue closure is claimed.

[yifanxuaaa]

CI budget update applied (owner-authorized150seconds)

Source: f35e003

Run34789404326 at7a4a920ac completed the native test selections in125seconds, then failed the original120second enclosing limit. Its failure is preserved unchanged: https://github.com/Ephemeral-AI-Lab/layerfs/blob/f35e0039ba8f11fb97feb608b2bc179fc0e9bc04/docs/roadmap/0.1/0.1.6/evidence/ci/run-34789404326-failed.log . The new budget applies prospectively; it does not relabel that attempt or change any correctness/benchmark contract.

Work continues on Host FUSE/SDK ownership, exact snapshot/Commit retry, bounded canonical construction and safe cleanup. No timing optimization detour, phase-completion claim or benchmark launch. Generic dirty-mmap acquisition remains an open supported-surface obligation. #124/#125 stay open.

[yifanxuaaa]

Owner-requested bounded checkpoint complete; source published to remote main

The owner changed this run's stopping point to finishing the three current integration fixes, publishing to main and leaving a successor handoff. This checkpoint does not complete a whole implementation-plan phase or the Snapshot-Isolated Workspace feature.

Changes and verification:

• Mounted host shutdown now releases its preopened mount-root descriptor before unmount. The full failing mounted component case now PASSes SDK mapping/descriptor visibility, unrelated dirty mapped bytes, inode/hardlink/rename/open-unlinked behavior, explicitly owned C1/C2 and normal helper unmount/container cleanup. PASS receipt; original EBUSY failure.
• The2MiB capacity fixture now uses deterministic varied input, forcing actual spill. Unchanged ownership/return-to-baseline assertions PASS. PASS; original non-spilling fixture failure.
• The maintenance oracle now proves a real retained canonical stage after publication failure. Its targeted rerun PASSes; the other2 passes are retained with dependency-specific justification. Manifest.

Source: verified main merge 74b2bdb93, merged PR#126. Main ancestry includes product/evidence commit f8ed6bb and budget/handoff head820ecab8079c7e0f09c29c6df1c983366e3dd77e. Native mounted binary SHA1f84e1f00c529a5796029a5fdfe644ac0faca06c4534353e84843f0993994325; Linux helper SHA29f56dcd24aad94149510d8f0474a2d9895b96ea7904465404aa39172a33e513. Exact image/source/limits and reproducible runner are linked in the receipt.

Publication validation: workspace build, formatting/whitespace,7 runner tests and executed native tests pass. Full native gate reported611 passes/26 ignored from637 selections,37 binaries,79 batches; its171s duration failed the then150s gate. The owner subsequently accepted171s and explicitly set180s, now delivered. Historical125s/120s and171s/150s failures remain unchanged. Strict Clippy remains FAIL and is documented; no lint suppression or required review/branch-rule bypass. Independent scoped review found no new blocker.

Durable successor handoff includes source/evidence custody, all retained failures, current phase, local state, resource ownership, CI debt and next executable task. No owned process/container/measurement lock remains.

Remaining: generic V1 acquisition, cohesive public/default FUSE/SDK/Commit migration, V2 capacity/accounting, full lifecycle/uncertainty acceptance, million-file qualification, final candidate sealing and the complete non-#122 benchmark campaign. Host-only captured input in the mounted check does not prove generic dirty-mmap acquisition. #124/#125 stay OPEN; all36 exact#122 cases stay excluded. No release/tag. Execution stops at this owner-requested checkpoint.

[yifanxuaaa]

Execution takeover: remaining Snapshot-Isolated Workspace work starts now.

Checkout and source actually being used

Item	Value
Workspace	/Users/yifanxu/Ephemeral-AI-Lab/layerfs
Branch / HEAD	main == origin/main at 3f04d4146ebe508a240076e4540de5123b9142d1 (PR #127 balanced Clippy merge)
Ancestry verified	f8ed6bbbb (product/evidence) and 820ecab80 (180s budget/handoff) are ancestors of HEAD; no local divergence, working tree clean
Governing documents	docs/roadmap/0.1/0.1.6/ rule, spec, spec-review, architecture, implementation-plan, contract-resolution, progress, verification ledger, exclusion manifest
Other worktrees	layerfs-transition-experiments, layerfs-transition-report (detached, unrelated) — not touched

No build, test, container or measurement-lock holder is running; the measurement lock file is unheld. This run does not reset, switch or discard the shared checkout.

Retained valid evidence

• All three bounded checkpoint fixes stay PASS with their original failures preserved: mounted shutdown (attempt01 EBUSY failure → attempt02 PASS), actual-spill capacity fixture (attempt01 FAIL → attempt02 PASS), maintenance oracle (attempt01 2 PASS/1 FAIL → attempt02 PASS). Reused without rerun; none of the source they exercise has changed.
• Component passes are retained under their recorded binary/seal identities: owned host operation core, canonical correspondence, retained-stage/publication receipts, SDK actor/lease/cancellation, Linux helper and codec builds, balanced-Clippy policy evidence.
• Historical gate timings stay labeled as failures: 125s/120s and 171s/150s. Current CI budget is 180s. No timing, oracle or benchmark contract is changed by this run.
• V1 probe and its retained-page counterexample remain the current mechanism evidence; they are not being relabeled or rerun.

Unresolved obligations (unchanged by this comment)

• V1 generic FUSE visibility of writable mappings (Commit acquisition) — OPEN; requires a distinct bounded hypothesis, not a rerun of the same probe.
• V2 ownership/capacity — the default max_payload_owners = 8192 admission counts persisted payload TOKEN rows, not transient handles, which predicts a false ceiling at the 8,193rd independently retained ordinary payload token. Reproducer NOT_RUN, no fix landed.
• Canonical scratch/admission accounting (provisional 96 MiB reservation, admission-owned seen spill, full-attempt peaks, scoped cleanup retry) — OPEN.
• V3/V4 — component checks pass; full public-path integration, recovery/End/Discard and authoritative unknown-publication resolution remain.
• Default FUSE/SDK/lifecycle migration to one host authority — OPEN; the ordinary public Workspace mount/Commit route is still legacy. No fallback was introduced.
• Million-changed-file public-path proof and candidate sealing — NOT_RUN.
• Full included benchmark campaign excluding the 36 exact v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 cases — NOT_STARTED.

Next concrete implementation task (executing now in this run)

Reproduce the persisted-token admission defect, then repair it at its shared source:

1. Add one focused regression through the existing host_overlay::tests::Fixture under the unchanged ResourcePolicy::default(): 8,193 distinct regular files in one HostOverlay, deterministic one-byte ordinary write per file, one bounded existing host maintenance step per completed write, no Commit, no per-file retained snapshot, no resident NodeId map. Record the first failure's rejected operation, token/handle/queue counters and charged storage; on rejection verify the rejected file stays empty and prior acknowledged bytes remain readable.
2. Separate persisted payload-token ownership (governed by the existing explicit catalog/physical disk quotas) from transient resident handles plus their queued release tickets (governed by the existing resident owner cap). Do not raise 8192, do not introduce an unbounded release queue or drain, and keep the existing State→released lock order.
3. Add bounded admission assistance so retired transient handles are processed outside the state lock rather than becoming a permanent ceiling; serialize with the existing maintenance mutex.
4. Verify a tiny configured owner cap still produces a safe quota error for genuinely simultaneous handles, then rerun only the original failing reproduction plus the smallest affected retained ownership/reclamation checks.

This task does not depend on V1. Step 2 (production FUSE/SDK integration) and Step 4 (generic FUSE visibility) remain queued behind it; the full million-file proof and the benchmark campaign stay gated on the correctness exits above. Both issues stay OPEN; #122 stays excluded; no release or tag.

[yifanxuaaa]

Step 1 progress: persisted-token admission defect reproduced, diagnosed, and repaired

The predicted defect was real but not the first thing that fails. The observed first failure was earlier and had a different cause, so I am recording both.

Observed failure (preserved). 8,193 distinct regular files in one HostOverlay, one deterministic one-byte ordinary write per file, one bounded HostOverlay::maintain() per write, ResourcePolicy::default(), no Commit and no per-file snapshot. File 6,841 was rejected with StorageFull: "payload catalog recovery reserve" (owners=6841 persisted_tokens=6841 pending_releases=0 physical=28020736 payload_index_physical=1082957824). The rejected file stayed empty and all 6,840 prior acknowledged bytes were readable, so this was a clean rejection, not corruption. The binding constraint was the payload catalog, not the 8,192 owner counter. Raw log: docs/roadmap/0.1/0.1.6/evidence/step1-capacity/attempt01-before-fix.log.

Root causes (measured).

1. Payload::reclaim_step never invoked Index::reclaim on the payload index, so retired page versions accumulated for the life of the Workspace. Isolated diagnostic: 200 one-byte writes retained 1,966 live index pages; the same run with a bounded recycle settles at 399. Reclaimable, not leaked.
2. Index::reclaim spent its whole budget on the retired-root queue before the zero-reference reclaim chain. Ordinary writes keep that queue continuously non-empty, so the chain barely advanced. Compounding this, the caller loops stopped on "no pages freed in this step" — but one chain step releases a child reference and frees no page, so a still-advancing cascade read as completion. The fixed 16-page HostOverlay::maintain() budget was also below the retirement rate of ordinary writes.

Repair (shared source, no cap raised).

• Resident ownership and persisted tokens are now distinct quantities: State.owners counts live OwnedRange handles plus their queued release tickets/jobs; State.tokens (Stats::persisted_tokens) counts live payload TOKEN rows and is charged to the independent payload-index/arena disk quotas. A dropped handle whose token is still owned by a metadata leaf frees its resident slot immediately.
• Resident admission is unchanged in strictness for genuinely simultaneous handles; max_payload_owners = 8192 was not raised. It is documented as a resident cap, not a changed-file ceiling.
• Bounded admission assistance (Payload::assist) recycles the catalog and retires queued handles before the state lock, serialized on the existing maintenance mutex, so no external maintenance loop is required for either backlog to keep up.
• Index::reclaim always advances the zero-reference chain and reports how many retired roots it consumed; Index::reclaim_pending gives a cheap backlog probe; the bounded drive loops now stop on empty backlog rather than on zero pages freed.

Verification.

• Tiny-cap oracle tiny_owner_cap_bounds_resident_handles_not_persisted_tokens (cap = 4): four simultaneous handles admitted, the fifth rejected with StorageFull while admitted owners remain usable; 64 independently retained tokens admitted at peak_resident=3 with a 119-page catalog; dropping the retaining root converges to persisted_tokens=0, owners=0, index_live_pages=0, cleanup_pending=false. PASS.
• Index invariant repeated_prefix_insertions_release_every_retired_page: pages reachable from the live root equal live pages after a drained reclaim. PASS.
• Original reproduction after repair: PASS — files=8193 verified_bytes=8193 peak_resident_owners=0 peak_pending_releases=0 persisted_tokens=8193 owners=0 pending_releases=0, payload index physical 71,593,984 B (≈1.97 pages/file, down from ≈16.2), arena 33,611,776 B. Command: cargo test -p layerfs-workspace --all-features --locked many_ordinary_files_do_not_exhaust_payload_owner_admission -- --ignored --nocapture --test-threads=1.
• Affected regression set: full layerfs-workspace suite serially = 162 + 12 + 2 passed, 0 failed, 13 ignored. (A single changes::tests::tiered_spill... failure under --test-threads=4 is an FD-race artifact of that test's process-global FD counter; it passes in isolation and serially, and is not caused by this change.)

Evidence: docs/roadmap/0.1/0.1.6/evidence/step1-capacity/README.md plus raw logs in the same directory.

Reused passes: the three bounded checkpoint fixes (mounted shutdown, actual-spill fixture, maintenance oracle) and the SDK/actor/canonical-correspondence/receipt passes do not exercise the modified payload-index, reclaim-budget or resident-accounting functions and were not rerun.

Remaining Step 1 obligations (not claimed complete): snapshot-reader cache accounting; canonical construction/admission reservations; concurrent Workspace admission; workspace-owned scratch placement; admission-owned spill lifetime; full-attempt peak sampling. The million-changed-file proof remains NOT_RUN and requires its own frozen fixture and quota plus the public-path/V1 exits — the 8,193-file case above is a scale regression, not that proof. #123 stays open.

Next executable action: proceed to the production FUSE/SDK default-path migration (Step 2) while a bounded parallel investigation establishes the V1 generic-mmap question, then Step 3 public Commit integration.

[yifanxuaaa]

Step 1 progress (continued): snapshot-reader cache accounting now charged, not assumed

Second Step 1 capacity obligation closed.

Defect. Every Workspace owns one ordinary SnapshotReader whose demand cache is independently bounded at 8 MiB by the Store (SNAPSHOT_CACHE_BYTES), but the host aggregate reservation never charged it. A Workspace's real resident footprint therefore exceeded its own reservation by up to that allowance, and cross-Workspace admission arithmetic silently omitted it. The Store's receipt fields exposed the cache size but nothing admitted it.

Repair. The bound is exported from the Store as SNAPSHOT_READER_CACHE_BYTES and included in the components() fixed resident base for one Workspace, so it is reserved through the existing Scheduler/HostAdmission path rather than assumed free. No policy value changed and no admission limit was relaxed.

Verification. New check fixed_resident_base_charges_the_snapshot_reader_cache: fixed base = 14,234,624 B for an 8 MiB reader cache under the 128 MiB aggregate allowance, and the check asserts the aggregate still admits several Workspaces rather than being silently consumed by one. PASS. Full layerfs-workspace + layerfs-layerstack-store suites serially: 163 + 12 + 2 + 163 + 8 + 1 + 1 passed, 0 failed, 13 + 6 + 8 ignored. Raw log: docs/roadmap/0.1/0.1.6/evidence/step1-capacity/suite-after-reader-cache.log.

Source. ff7098929 on PR #128 (with the earlier ca46e2793 payload-ownership repair).

Reused passes: the payload-ownership reproducer, tiny-cap oracle and index invariant from the previous comment remain valid — this change touches only the fixed resident base constant and the Store's exported bound, not the payload/index reclamation paths. The three bounded checkpoint fixes and all SDK/actor/correspondence/receipt passes were not rerun.

Still open in Step 1 (not claimed complete): canonical construction/admission reservations and the full-attempt peak envelope; concurrent Workspace admission behavior under real loads; Workspace-owned scratch placement; admission-owned seen-spill lifetime; bounded cleanup/reclamation integration into runtime End. The million-changed-file proof remains NOT_RUN under the unchanged #123 contract.

Next executable action: production FUSE/SDK default-path integration (Step 2) from the consumer audit, while the bounded V1 generic-mmap investigation continues in parallel.

[yifanxuaaa]

Step 1 progress (continued): Workspace-owned scratch placement

Third Step 1 obligation closed.

Defect. Private canonical scratch was created in the shared process temporary directory (std::env::temp_dir()), so a Workspace's canonical construction scratch had no owning placement and no scoped custody or cleanup path.

Change. ScratchBudget::with_directory(limits, owner, Option<PathBuf>) adds an explicit placement directory while new keeps its existing meaning. An explicitly configured directory must already exist — a missing placement is rejected rather than silently redirected, and no ambient context can redirect private scratch. Scoped spill output now creates its named file through the scope's directory, and CandidateCapacity::acquire_in(budget, policy, placement) takes the owning Workspace spool directory, which snapshot_candidate supplies.

Custody preserved. Named allocations remain identity-bound, unlinked on drop, 0600 and single-link; descriptor and named-owner charges are unchanged.

Verification. named_scope_scratch_is_placed_in_the_owning_directory (path parent is the owning directory, mode 0600, nlink == 1), explicit_scratch_placement_rejects_a_missing_directory, and the candidate-capacity placement assertion all PASS. Full layerfs-layerstack-store + layerfs-workspace suites serially: 163 + 161 + 12 + 2 + 8 + 1 + 1 passed, 0 failed. Raw log: docs/roadmap/0.1/0.1.6/evidence/step1-capacity/suite-scratch-placement.log.

Source. 737592356 on PR #129. The earlier capacity repairs are merged to remote main at 58d25702195df3226443b4f617f995264a81e7be (PR #128), and its CI run 34800385903 passed.

Reused passes: the payload-ownership reproducer, tiny-cap oracle, index invariant and reader-cache charge checks are unaffected by scratch placement and were not rerun; the merged capacity repair's CI result is reused as-is.

Still open in Step 1 (not claimed complete): canonical construction/admission reservations and the real full-attempt peak envelope (the 96 MiB draft is an analysis figure, not a verified minimum); concurrent Workspace admission under real concurrent load; admission-owned seen-spill lifetime; bounded cleanup/reclamation driven from runtime End. The #123 million-file proof remains NOT_RUN.

Parallel work in flight: the production host-authority migration for the ordinary public route (Step 2) and a bounded distinct-hypothesis investigation of generic FUSE mmap snapshot visibility (Step 4/V1). Neither is complete; no phase is claimed.