v0.1.6: spill pending Workspace metadata to disk for bounded-memory large change sets

[yifanxuaaa]

Objective

Make the existing base-plus-overlay Workspace model support large current uncommitted states with bounded RAM. Move pending file/range metadata to temporary disk backing as needed, keeping a bounded memory cache. Preserve one explicit Commit boundary even when many files change; spilling must not publish intermediate commits.

This is v0.1.6 product work, related to the benchmark campaign in #122 and the pending-capacity work in #116. Reuse the existing Workspace, backing, and spill mechanisms where suitable; this is not a rewrite of CAS, packing, compression, or the Commit storage engine.

Problem and clarified semantics

The Workspace represents current uncommitted state, not an operation log. Commit captures the resulting state. Repeated edits replace superseded state, while changes to distinct ranges/files must remain represented if they still contribute to the result.

Current pending-piece allocation is capped at 2 MiB across a Workspace. The v0.1.5 qualification of 32,000 edits means 32,000 distinct files each holding one eligible 64-byte overwrite descriptor: 2,048,000 bytes. It is not a 32,000-action limit or a general file-count limit. A million such descriptors would require approximately 61 MiB before payloads and other metadata.

Current compact forms specialize one equal-length inline/spool overwrite; further edits or unequal-length replacements use the general piece tree. The edit counter is informational. Ordinary write payloads already use spool storage, but pending metadata remains subject to the memory ceiling. Superseded spool allocations can outlive their logical replacement.

Unchanged files must not acquire an edit descriptor merely because they exist. Per-inode pending state is charged in aggregate across its Workspace; namespace, payload, and other resources remain separately accounted.

Implementation scope

• Keep committed content immutable and maintain one authoritative current overlay entry per changed inode, preserving hardlink identity and existing namespace semantics.
• Make pending metadata spillable with bounded resident memory. Reaching a cache allowance should evict/spill eligible state rather than reject an otherwise valid edit solely because metadata no longer fits in RAM. Keep explicit temporary-storage quotas and safe errors for actual resource exhaustion.
• Reuse existing disk/backing/index facilities after tracing the actual read/edit/Commit paths. Choose the smallest implementation that supports the required scale; do not prescribe a new database or dependency prematurely.
• Support equal- and unequal-length edits, insert/delete, append/truncate, large files and large spool-backed writes through consistent range semantics. Compact representations may remain fast paths, but supported capacity must not depend on the equal-length specialization. Preserve and document public input limits such as SDK inline limits.
• Update current state instead of retaining a per-action history. Coalesce compatible ranges where useful and reclaim superseded metadata/payload storage when no current state, retained reader, or in-flight operation needs it. Define an observable bound on deferred reclamation.
• Preserve atomic edit installation, retained-reader correctness, FUSE/SDK coherence, Commit snapshot/publication boundaries, Discard and cleanup. Spill/eviction is not Commit. Failed spill or quota admission must preserve the prior valid state.
• Account pending metadata residency, cached/spilled entries, inline bytes, logical/physical spool bytes, pinned state and cleanup separately. A configurable memory budget alone is an intermediate improvement, not completion of disk-backed scale support.

Acceptance criteria

• Record the chosen design and resource/ownership invariants, including how all resident indexes and pinned/in-flight entries are bounded; moving only the payload of an unbounded in-memory map is insufficient.
• Prove spill and reload through public Workspace read/edit/Commit paths under a deliberately small metadata-cache allowance. Verify exact content and metadata before Commit and after Commit plus fresh Store reopen.
• Qualify an explicitly selected scale case with at least 1,000,000 distinct changed regular files in one Workspace before one final Commit, using a fixed declared RAM allowance and adequate explicit disk quota. Report total files separately from changed inodes and distinguish pending-metadata residency from total process RSS. Do not substitute incremental commits or several Workspaces.
• Include a large unchanged namespace with a small changed subset to demonstrate that unchanged files are not charged as pending edits; report remaining namespace costs honestly.
• Exercise fixed-work repeated same-range/whole-file overwrites and disjoint edits. Show obsolete state is reclaimed within the declared policy, rather than growing with historical action count after readers release references. Verify the current result independently.
• Cover unequal-length edits, boundary-sized and large files, spool-backed writes, hardlink aliases, rename/unlink and open-unlinked lifetime. Include held readers across edit, spill/eviction and reclamation.
• Inject quota exhaustion and backing I/O failures at state transitions. Verify failed operations leave the previous state readable, publish no partial Commit and clean up temporary resources safely.
• Check concurrent Workspace isolation and Commit/Discard cleanup using existing tests and v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122 infrastructure where applicable.
• Publish matched before/after small-workload latency and explicit scale measurements: edit/read/Commit time, host RSS, pending cache/index/pinned bytes, spill I/O, temporary disk growth and reclamation. Freeze fixtures, budgets and acceptance thresholds before collection; retain misses and failures.
• Update Workspace limits/documentation to distinguish actions, live pieces, changed files/inodes, RAM allowance, disk quota and public input limits. Remove the implication of a universal 32,000-edit ceiling.

The million-file case is an explicit scale qualification, not an unannounced addition to #122's regular 15-second matrix. Declare its fixed work, deadline, fixture and verification coverage before running it. Do not claim million-file support from extrapolated descriptor arithmetic alone.

Non-goals

• No new crash/power-loss durability or restartable-Workspace promise merely because temporary state resides on disk; durable Commit/recovery remains separate (v0.3.0: Durable Commit and explicit fsync/recovery guarantees #69).
• No replacement of CAS/CDC, persisted FULL/DELTA encodings, pack compression or canonical authentication.
• No reintroduction of explicit Store compaction, automatic intermediate commits, unbounded caches, or removal of resource validation.

Starting points

• crates/layerfs-workspace-core/src/file_edit.rs: compact forms, general piece tree, aggregate resource accounting and informational edit counter.
• crates/layerfs-workspace-core/src/lib.rs: Workspace resident state and counters.
• crates/layerfs-workspace/src/file_io.rs, live_backing.rs, changes.rs: writes, backing ownership, pending state and Commit integration.
• crates/layerfs-workspace/tests/issue116_capacity.rs: existing capacity probes; distinguish historical probes from current behavior.
• docs/releases/v0.1.5/CHANGELOG.md and docs/roadmap/0.1/0.1.6/: released capacity evidence and current qualification plans.

At issue creation this is requested work, not an implemented or qualified architecture change.