Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks

[yifanxuaaa]

Promoted migration: existing tiny-churn evaluation

Latest owner direction supersedes the earlier 25k/two-second milestone: target close performance to the existing tiny- benchmarks, reject quadratic scaling, and defer the 25,000-file and million-file cases.* #130 remains promoted into the active migration; it does not wait for #124/#125 closure.

The current implementation plan governs the selected architecture, iteration order, comparison and acceptance. This is reviewed planning, not implemented performance. The former 25k case registration and two-second gate are deferred. Million-file qualification remains DEFERRED/OPEN wherever the broader migration requires it; this does not waive or satisfy that obligation.

Successor execution prompt

Use the published execution handoff to implement and verify the full current P130 rollout. It requires actual package-completion comments, five tier-500 cases then all 20 when stable, retained failure/pass validity, reviewed main delivery and verified terminal #130 closure. No stop at a replan, revision, one green case or subagent completion while ready authorized work remains. Genuine external blocks remain explicit and never become PASS.

The owner also prohibits patching, forking, vendoring, replacing or copying/reimplementing third-party libraries/imported implementations, and changing third-party dependency/import declarations to bypass an API limitation. Use existing documented APIs and first-party code; no dependency forks/overrides/new packages/version changes. See the handoff's precise boundary and external-block handling.

Quick iteration: exactly five tier-500 cases

Case	Actual workload
tiny-create-500-mixed-v4	500 created tiny files with the registered mixed background
tiny-stat-500-mixed-v4	500 stat targets with the registered mixed background
tiny-unlink-500-mixed-v4	500 removed tiny files with the registered mixed background
tiny-bulk-create-500-mixed-v3	5,000 created files / 500 MiB plus registered witness
tiny-bulk-delete-500-mixed-v3	5,000 removed files / 500 MiB plus registered witness

The bulk 500 tier does not mean 500 files. Preserve fixtures, byte distributions, metadata normalization, root sync, worker counts, seeds, timers and independent verification. Do not substitute smaller cases, one-byte workloads, SDK bulk operations or the legacy authority route.

Establish results for these five, then iterate on the failing case and genuinely affected regressions only. Reuse valid passes across unrelated edits. When stable, qualify all 20 existing tiny_file_churn cases; the five-case quick screen is not full-family or full #125 completion. All 20 are outside the 36 #122 exclusions.

What “close performance” means

Use a compatible v0.1.5 control and the existing #118 ordinary paired regression screen, prospectively frozen for this evaluation:

• Existing n3 fresh alternating pairs, retaining its arm/seed/custody rules.
• A material wall regression requires median paired slowdown > max(15% of control median, 3 ms) AND at least two of three pairs slower.
• CPU analogue: max(15% of control median, 1 ms).
• Stronger applicable criteria prevail, including registered strict <1 s tier 100 bulk create/delete and tiny-create-100 criteria where applicable. Correctness/resource/cleanup/deadline failures cannot be excused by a relative screen.

Apply the screen to each registered whole-workflow metric; receipt operations=1 must not be divided by 500/5,000 files to dilute the noise floor. Preserve the inherited pure_call_sum, and report Begin/Exec/Commit/visibility/End and orchestration wall separately. No universal two-second target or new per-file threshold.

Historical tiny-create-500 measured 242.659707 ms for 500 files / 824,450 B on a 5,000-file/ 500 MiB background; bulk-create-500 measured 5,732.993668 ms for 5,000 files/ 500 MiB, including large files. Those are context until a matched control is established. If compatible public execution cannot be established, comparison is unavailable/blocked, not “close” from unlike routes. Preserve historical WARN/FAIL/waiver/reuse labels.

Explicit issue ownership

Issue	Owns
#130	Implement overhead reduction: local payload reclamation, bounded Index preparation, measured compact storage/correspondence/lifecycle/transport changes, and their five/20-case tiny-churn comparisons and affected correctness checks
#124	Complete production macOS-host/Docker FUSE and SDK wiring, V1 snapshot semantics, Commit integration, non-Commit consumers and obsolete-coupling removal
#125	Later full included benchmark campaign, final matrix/custody/report and its terminal requirements

Promoting #130 means implementing its own changes now alongside #124. It does not rename #124's unfinished integration or make complete #124 success a prerequisite for the first optimization. Authentic public timing depends on the verified new route; independent allocator/Index/range/correspondence work can start immediately with owned inputs. The interrupted four-file #124 patch remains separate.

First #130 implementation slice: reproduce and repair quadratic arena reclamation, then remove redundant Index/range preparation. Comparator/instrumentation work supports these code changes; it does not replace them.

Architecture review and #130 implementation order

Two subagents reviewed existing tiny-churn receipts/registry and actual write/Index/payload/transport source, then reassessed the plan for this revised scope. The coordinating review checked governing contracts.

1. P130.1 — Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130 comparator and cost instrumentation. Freeze the five tier-500 comparisons and instrument the actual overhead. Consume v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124's verified production route for public timing. If that route is pending, only dependent public measurements wait; proceed with independent Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130 component implementation/checks. Record exact release source/binary/image/fixture identity and separate transport/host/Index costs.
2. P130.2 — Repair quadratic reclamation and redundant preparation. Current release-triggered whole-arena evacuation can move N+(N−1)+… live blocks across small deletions with completed maintenance: quadratic cumulative work. Replace it with indexed free-space reuse/local tail-block reclamation and an amortized proof. Prepare bounded final operation records instead of repeatedly constructing/retiring intermediate Index roots.
3. P130.3 — Select the next measured storage cost. Use denser existing Index pages, Empty/Single/Indexed ranges, packed tiny allocation in existing arenas or metadata-only singleton correspondence only where the case measurements justify it. The full compact-storage bundle is not a prerequisite for the first useful result.
4. P130.4 — Remove measured remaining latency. Lazy unused resources, bounded immutable-page cache or a correct transport change are conditional. Serialized host requests are an independent cost; preserve host ownership before acknowledgment and exact replay. No mandatory RAM-first pager or replacement backend.
5. P130.5 — Qualify Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130 changes and hand off. Five quick cases first, then all 20 when stable, plus genuinely affected correctness/resource/cleanup checks. This verifies the delivered overhead implementation. v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124 still owns unfinished V1/production/consumer migration; v0.1.6: run full existing benchmarks excluding #122 and report actual numbers #125 owns the later full matrix. Link required cross-issue evidence without relabeling that implementation as Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130.

No quadratic scaling is acceptable, including deferred maintenance. Linear necessary input/output processing is allowed; prefer logarithmic point/index operations and O(1) snapshot ownership. D logarithmic updates may total O(D log N); disclose those factors. Fixed-size maintenance calls are not proof of bounded cumulative work.

Required guards

Keep the single host authority, immutable published metadata, exact leased-source installation, independent file/snapshot ownership, bounded journals/admission and existing canonical construction/CAS/authentication/FULL-DELTA/CDC/extents/packing/compression/staging/publication machinery. A file-only reader must not retain unrelated payload tokens through a shared leaf. Packed physical slack remains bounded and charged; preserve append/Origin/promotion, fsync, relocation, rollback and release headroom.

V1 dirty-writable-mapping visibility is still open. No host-root-only full-surface claim, implicit user fsync, removed mmap, third-party patch, global freeze/drain, remount or checkpoint reset is authorized. Live operations/handles must survive Commit and retained stages; C1/C2 and lost Created/UpToDate resolution stay exact. Preserve original failures and the interrupted unverified Step 5 patch rather than claiming it passed.

Acceptance and execution discipline

• Verify/freeze the five exact quick cases, comparator, existing criteria and actual new production dispatch.
• Repair quadratic cumulative reclamation with focused source-bound proof and affected regressions.
• Remove measured redundant work; select and version additional representation changes only for demonstrated costs.
• Achieve close per-case performance under the existing screen and all stronger applicable criteria; no required failure hidden by a family average or inner Commit time.
• Qualify all 20 inherited tiny-churn cases when stable, preserving exact fixtures, seeds/repetitions, independent proofs and cleanup.
• Publish real time, CPU/RSS/cgroup, metadata/payload/index/retained/scratch/slack storage and I/O. Unavailable is not zero; earlier 105–245 MiB estimates are not accepted per-call allowances.
• Preserve affected natural-overlap, C1/C2, large-file and supported FUSE/SDK checks. Keep existing 100k-base D=0/1/10 and 100k-changed accounting obligations explicit; neither is a quick-iteration prerequisite or a substitute for deferred million-file qualification.
• Retain 25k/two-second and million-file cases as deferred, neither passed nor waived. Do not register/run them in this evaluation.
• Publish exact source, retained failures/fixes and final applicable evidence before claiming completion. Broader v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124/v0.1.6: run full existing benchmarks excluding #122 and report actual numbers #125 closure remains subject to their still-open terminal obligations.

Post a substantive source/evidence/failures/reused-passes/next-action comment after each actual P130 completion; mirror production/correctness to #124 and benchmark-facing changes to #125. Original seven-phase completion comments remain required. During implementation, bounded subagents iterate on concrete checks and integrate results; no stop at a revision or one green test while ready authorized work remains. Reuse valid passes and existing facilities. Do not rerun the whole five- or 20-case set after unrelated fixes.

Use the existing runner and measurement lock. macOS owns Store/SDK/coordinator/construction/spool; Linux Docker owns daemon/FUSE/workload under existing 2 CPU / 2 GiB / no-swap / 256 PID limits. No concurrent resource-sensitive work. All 36 #122 cases (33 regular + 3 extended) remain excluded; no shared/v016_matrix.py campaign, release/tag, website deployment or #123 closure.

[yifanxuaaa]

#130 promoted: tier-500 tiny-churn iteration and non-quadratic architecture

Latest owner direction supersedes the earlier 25k/two-second milestone: quick iteration targets the five existing tier-500 cases below, seeking close performance under the existing paired regression screen and stronger case criteria. The 25,000-file and million-file cases are deferred, not passed or waived. Do not register or run them for this evaluation.

• tiny-create-500-mixed-v4
• tiny-stat-500-mixed-v4
• tiny-unlink-500-mixed-v4
• tiny-bulk-create-500-mixed-v3
• tiny-bulk-delete-500-mixed-v3

Bulk tier 500 means 5,000 files / 500 MiB; the other three cases affect 500 files. Preserve fixtures, normalization/root sync, timers, seeds and independent proof. Establish results for the five, then rerun only failing or genuinely affected checks. Full 20-case tiny-churn qualification follows when stable, not after every fix.

Current immutable implementation plan · Review PR #135

Two subagents reviewed benchmarks/registry and the actual write/storage/transport path, then reassessed the plan for the new scope. Repair release-triggered whole-arena evacuation, which can produce quadratic cumulative relocation, and redundant Index preparation first. Select density, compact ranges, packed arena allocation, correspondence, caching or transport changes only for measured remaining costs; a complete storage bundle or RAM-first pager is not mandatory.

This is documentation/review publication, not a P130 package completion or performance PASS. Document scope/links/whitespace were checked, and the interrupted four-file product patch remains byte-identical. No new product build/test, benchmark or V1 probe was run. V1–V4, supported non-pausing behavior, ownership and publication remain required; #122 remains excluded. Existing broader capacity/closure obligations stay open while deferred.

[yifanxuaaa]

Execution handoff delivered to clean main; branch cleanup verified

The owner-requested cleanup is complete. Local main and fetched origin/main both point to 74a62398485ce2cb6a452f7adf8f96dfb64e989b; the working tree is clean, main is the only local branch, there is one worktree and no remaining stashes. Completed/archived migration branches were removed after preservation. Unrelated open PR branches #36/#37/#119 were preserved as requested.

Source: merged handoff PR #136, following merged plan PR #135. Executable successor prompt. Preservation inventory.

CI 34810718677 passed, and the delivered main tree equals the tested PR tree. The unfinished four-file Step 5 patch, trajectory logs, cancelled probe evidence and historical ledger rows are preserved without enabling unverified product changes. A verified external Git bundle also preserves original branch/stash/worktree history.

The handoff requires actual P130 implementation and verification through the five tier-500 quick cases and all 20 when stable, completion comments for every actual package, valid failure/pass reuse, reviewed main delivery and verified terminal #130 closure. It prohibits third-party library/dependency/import patches, forks, substitutes and copied reimplementations. Genuine external blocks remain explicit, never PASS; 25k/million work remains deferred and #122 excluded.

This completes cleanup and handoff publication only. No P130 implementation package or benchmark campaign is claimed complete; #124/#125/#130 remain OPEN.

[yifanxuaaa]

Takeover: P130.2 local payload reclamation (execution run)

Source. main = 74a62398485ce2cb6a452f7adf8f96dfb64e989b (= freshly fetched origin/main, clean tree, only local branch main). Working source of record for this slice: crates/layerfs-workspace/src/overlay_payload.rs plus its focused tests. No dependency/manifest changes are authorized or planned.

Retained evidence (not re-run, not invalidated). docs/roadmap/0.1/0.1.6/evidence/minimal-overhead-review/ (review + manifests + preserved Step 5 archive inventory), evidence/step1-capacity/ (25k diagnostic, retained as-is), evidence/v1-*, evidence/step5-non-commit-consumers/ (UNVERIFIED patch, not applied), evidence/step2-host-integration/. The historical 25k/two-second milestone and million-file qualification remain DEFERRED/OPEN, neither PASS nor waived; no deferred case is registered or run in this execution.

Open obligations carried into this run.

1. P130.2 first slice: reproduce the release-triggered whole-arena evacuation counterexample in a focused check, then replace it with indexed free-space reuse + tail truncation + bounded tail relocation with an amortized work argument.
2. P130.2 second half: prepare an operation's final Index/range records together (small range leaf first); no intermediate root churn.
3. P130.3/P130.4: select further storage/latency work only from measured cost evidence; record a decision when none is justified.
4. P130.1/P130.5: five tier-500 cases then all 20 tiny-churn cases under the prospective n3 paired screen. Public timing requires v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124's verified host-authority/Docker dispatch; if that route is not executable in this run the dependent public comparison is reported BLOCKED with exact evidence rather than substituted from legacy dispatch.
5. v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124 keeps production macOS-host/Docker FUSE/SDK wiring, V1 capture, remaining Commit/consumer migration; v0.1.6: run full existing benchmarks excluding #122 and report actual numbers #125 keeps the full included campaign. Neither is closed or relabelled here.

First concrete implementation task (starts in this same run). Reproduce and repair the quadratic reclamation in overlay_payload.rs: coverage release currently sets a sweep flag that evacuates the entire active arena (one 64 KiB step at a time), so D small deletions among N live blocks move Θ(D·N) live bytes. Replace it with (a) an indexed free-interval family in the existing payload catalog, (b) allocation that reuses a fitting free interval before extending the arena, (c) tail truncation, and (d) at most one bounded (≤ 64 KiB) relocation of the topmost live interval into a free interval when that strictly shortens the arena — with reader-range protection so no retained reader loses bytes. Exit for this slice: a focused test that counts relocated bytes for repeated deletions/replacements over a large unrelated live set, plus the existing payload ownership/quota/partial-I/O tests passing.

[yifanxuaaa]

P130.2 complete: local payload reclamation and prepared Index records

Changes: <implemented and removed/replaced>

• Removed the release-triggered whole-arena evacuation in crates/layerfs-workspace/src/overlay_payload.rs (State::{sweep,evacuating,moving,truncate}, the PHYSICAL-tail evacuation step, finish_truncate/trim_failed). It moved all live payload of the active arena (64 KiB per step) on every sweep flag, i.e. N + (N-1) + … live bytes for D small deletions with completed maintenance between them.
• Added indexed local reclamation in the same module: a FREE record family (key(FREE, arena, offset) -> [len], ascending offset, coalesced neighbours) inside the same immutable published root; coverage release at refs == 1 splits the LOCATION record at the released boundaries and registers exactly the freed physical span; allocation reuses the lowest fitting unclaimed interval for bounded payloads and otherwise extends the append tail; one bounded maintenance step truncates the top unclaimed interval or relocates one live interval (<= 64 KiB) into a strictly lower unclaimed interval. A relocation permanently removes at least the moved byte count from the arena length, so cumulative relocation is charged to reclaimed bytes. Live PhysicalLease ranges are stored explicitly (Reader { arena, start, end }), block any reuse of their bytes, and bound truncation; deferred work is reported through cleanup_pending and retained slack stays charged.
• Removed the per-field intermediate Index roots in the range-leaf preparation (overlay_ranges.rs::node): META/LEFT/RIGHT/PROVENANCE/BACKING are now prepared together through a new bounded Index::set_batch/insert_many (overlay_index.rs) that applies up to one leaf of records with one path copy and one leaf write, reusing the existing page allocation, external-owner, linked-child, retire and rollback transactions; value encoding is shared with set_record.
• Added instrumentation: Stats::{unclaimed_bytes, reused_bytes} and host_overlay::tests::tiny_create_metadata_page_cost_is_measured, which reports the measured per-tiny-file cost (create 62.9 page writes / 378 page reads, first write 21.7 / 126, payload catalog 8 allocated pages, payload 4 KiB per file).

Source: 9bfcbc768 + 0a4e55540 on branch codex/issue130-local-reclamation, PR #137, based on delivered main 74a62398485ce2cb6a452f7adf8f96dfb64e989b. No dependency, manifest, lockfile, [patch]/[replace], vendor or third-party edit; first-party crates only. No benchmark binary/image identity is claimed for this package (see Remaining).

Verification:

• RUSTUP_TOOLCHAIN=1.85.1 tools/test-fast.sh → PASS full workspace native tests in 157s with 4 bounded jobs (ceiling 180s).
• cargo fmt --all --check PASS; cargo clippy -p layerfs-workspace --all-features shows no correctness/suspicious lint.
• cargo test -p layerfs-workspace --all-features --lib -- --test-threads=1 → 171 passed, 0 failed, 13 ignored.
• Counterexample, identical workload/oracle on the pre-repair source (74a623984, temporary test applied to the stashed tree, then removed): 64 deletions among 64 live 8-KiB files, 524,288 B freed → relocated 33,030,144 B (63x amplification) and the amortized assertion fails.
• After the repair: same workload relocates 524,288 B = exactly the freed bytes; multi-size oracle (files=16/32/64) → freed 131,072 / 262,144 / 524,288 vs relocated 131,072 / 262,144 / 524,288, unclaimed 0, arena 524,288 + 57,344 retained slack at 64. Pre-repair source has no FREE family at all.
• Focused checks: repeated_small_release_over_large_live_set_moves_only_justified_bytes, reclamation_work_follows_freed_bytes_across_sizes, middle_release_reuses_unclaimed_interval_without_relocating_live_payload (zero relocation under a >64 KiB live top interval, gap reused in place, arena does not grow), live_physical_reader_defers_arena_truncation_without_blocking_writes (leased bytes stay readable and are neither reused nor truncated; a foreground write is admitted), prepared_batch_writes_the_same_tree_with_fewer_pages (5 → 1 leaf page writes, 2 → 1 allocated pages), plus the unchanged payload ownership/quota/partial-I/O/short-write/sync and range/overlay suites.

Reused passes: all unaffected layerfs-workspace, layerfs-fuse, layerfs-content, layerfs-daemon, layerfs-cli, fs-benchmark-pro unit/integration tests in the same suite run; the historical 25k diagnostic, V1 probe, Step 1 capacity and Step 5 archive evidence are untouched and remain valid for their original claims only.

Failures resolved: raw parallel cargo test runs show changes::tests::tiered_spill_partial_writes_and_final_merge_are_retryable_and_clean (descriptor-leak assertion) and four lifecycle::tests::host_authority_* failures (NoPermits) plus intermittent live_backing::tests::live_owner_builds_and_checkpoints_through_real_host_backing / overlay_budget::tests::two_workspaces_share_aggregate_reservation_and_failed_op_does_not_leak. All of them reproduce on unmodified main (verified by stashing this change), all pass with --test-threads=1, and the whole gate passes in 157s, so they are pre-existing parallel resource-interference flakes (same family as L37), not regressions from this package. No hidden retry-as-green and no lint suppression was used.

Remaining:

1. The public five-case/20-case comparison is BLOCKED on an v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124 integration dependency, not on this implementation: in the benchmark topology (macOS host + Linux Docker workload) the container placement mounts through crates/layerfs-workspace/src/projection.rs WorkspacePlacement::Container → live_backing::RemoteWorkspace::start + docker::DockerProjection::attach (the legacy remote protocol). install_host_runtime has exactly two call sites — the Host + Projection::Fuse path and tests — so live_backing.rs::generation()'s worker.host_runtime()? is None on the container route. Payload/Ranges/Index (this package's modules) are constructed only in overlay_budget::Resources::open for HostOverlay. A legacy-route case run therefore cannot qualify this implementation, and no case was run.
2. Next executable Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130 task (dependency-ready): the measured dominant per-operation cost is metadata path copies — create 62.9 page writes / 378 page reads, first write 21.7 / 126 per tiny file, with roughly half of the create's index operations being change-log bookkeeping (Mutation::changed = 1 get + 1 remove + 2 sets per key, 3 keys per create). Prepare those final records together through the same bounded set_batch mechanism.
3. v0.1.6: implement Snapshot-Isolated Workspace with non-pausing Commit #124 keeps production container-side host-authority wiring/V1; v0.1.6: run full existing benchmarks excluding #122 and report actual numbers #125 keeps the full included campaign. The 25k/two-second and million-file cases stay DEFERRED/OPEN; Promoted: minimize Workspace overhead against tier-500 tiny-churn benchmarks #130 stays OPEN.

[yifanxuaaa]

P130.2 addendum: prepared change-log records for ordinary operations (measured)

Changes: ordinary HostOverlay::mutate operations now opt into a bounded deferred change batch. Mutation::changed queues distinct change keys (deduplicated; capped at MAX_PENDING_CHANGES with an early flush), and Overlay::prepare applies them after the operation succeeds, removing superseded sequence keys first and writing up to BATCH_RECORDS prepared records per Index::set_batch call. forget_covered flushes first so a mutation that reads the change family can never observe stale deferred records; maintenance/cleanup mutations keep the immediate per-key behaviour. This continues the P130.2 deliverable "prepare an operation's final Index records together" beyond the range leaf.

Source: 12745110a (with 9bfcbc768, 0a4e55540) on branch codex/issue130-local-reclamation, PR #137. First-party crates only; no dependency/manifest/lockfile change.

Verification — measured page-write/page-read counters on the ordinary create + first-write path, 100 tiny files, baseline main 74a623984 vs this head (host_overlay::tests::tiny_create_metadata_page_cost_is_measured, deterministic across repeated runs):

phase	baseline writes	now	baseline reads	now
create_file	6,290	4,821 (-23.4%)	37,813	28,023 (-25.9%)
first write (1 byte)	2,165	2,068 (-4.5%)	12,642	11,037 (-12.7%)
total per 100 files	8,455	6,889 (-18.5%)	50,455	39,060 (-22.6%)

Per tiny file: create 48.2 writes / 280 reads, write 20.7 / 110. Payload catalog allocation is unchanged (802 pages per 100 files) and payload physical stays ~4 KiB per file.

Focused equivalence test overlay::tests::batched_change_records_match_immediate_tracking_with_fewer_page_writes: identical change lists, inode/binding/kernel-reference state, superseded-sequence replacement and later-mutation replacement against the immediate path, with 24 → 11 page writes for the same four edits.

Gates: RUSTUP_TOOLCHAIN=1.85.1 tools/test-fast.sh → PASS full workspace native tests in 149s with 4 bounded jobs; cargo test -p layerfs-workspace --all-features --lib -- --test-threads=1 → 173 passed, 0 failed, 13 ignored; cargo fmt --all --check PASS; no correctness/suspicious clippy finding.

Remaining unchanged from the previous comment: the five-case/20-case public comparison is BLOCKED on the #124 container-placement host-authority wiring (legacy remote mount path), no case was run, and #130 stays OPEN.