v0.3.0: Durable Commit and explicit fsync/recovery guarantees

[yifanxuaaa]

Problem and goal

LayerFS currently acknowledges live-process visibility without guaranteeing that an acknowledged snapshot survives an OS crash or power loss. v0.3.0 should introduce an explicit, tested durability contract, starting with durable public Commit: after Commit returns successfully, the published snapshot, its dependencies, and the corresponding history/head changes survive the supported crash and recovery scenarios.

Related: #52 (v0.3.0 cloud native integration and S3-compatible persistence). This issue owns the local durability foundation and its acknowledgement contract; coordinate remote replication semantics with #52.

Current behavior

• The Store configures SQLite journal_mode=MEMORY, synchronous=OFF, and locking_mode=EXCLUSIVE, and rejects WAL Stores during opening.
• Workspace fsync currently validates spool state and completes capture; it does not provide a disk persistence guarantee. Spool segment files are unlinked after opening, so adding sync_all() alone would not make live edits recoverable.
• A public Commit can include multiple object-admission transactions, a staged-root transaction, and final publication. Durability cannot be estimated or implemented as one unconditional fsync per public operation.

Relevant code: crates/layerfs-layerstack-store/src/schema.rs, objects.rs, workspace.rs, staging.rs; crates/layerfs-workspace/src/file_io.rs and lifecycle.rs.

Scope and acceptance

• Define acknowledgement guarantees for ordinary writes, application file/directory fsync, public Commit, initialization/Add, and remote replication. State which data and metadata each boundary protects, and distinguish process loss, OS crash/power loss, and loss of the storage device or host.
• Implement durable public Commit first. Persist all referenced canonical objects and required metadata before acknowledging the atomically published result; preserve conditional head updates, retained history, and v0.2 reconciliation semantics.
• Select the smallest recoverable SQLite journal/synchronization configuration. Evaluate WAL + FULL and an appropriate disk-backed rollback-journal alternative; do not merely change synchronous while leaving MEMORY journaling in place. Specify platform flush semantics, checkpointing, locking, Store creation/directory persistence, and migration/reopen compatibility.
• Handle I/O and synchronization failures without reporting false durable success. Define recovery and retry behavior when publication succeeds but its response is lost, and when the persistence outcome is uncertain.
• Make an explicit, separate decision on durable live-workspace file/directory fsync. If included, persist the recoverable edit representation and required metadata as well as bytes. If deferred, document the weaker uncommitted-workspace contract clearly; durable Commit must not imply that every successful application fsync is power-loss durable.
• Keep memory, dirty state, journal/WAL growth, and retry work bounded. Preserve canonical identities and protect published dependencies during recovery and cleanup.

Correctness proof

• Exercise failures before/after object admission, staging, publication, synchronization, and acknowledgement; cover synchronization errors and lost replies.
• Reopen after failure and verify database integrity, acknowledged roots, history/heads, canonical IDs, and file bytes. Never expose a published root with missing dependencies or silently lose an acknowledged durable snapshot.
• Prove migration and supported reopen behavior for existing Stores, including recovery with required journal/WAL sidecars.
• Distinguish process-kill tests from evidence for OS-crash/power-loss behavior. Freeze the supported platform/storage assumptions and retain reproducible failure evidence.

Performance validation

Measure the existing benchmark workloads under separately named profiles: current visibility-only baseline, durable Store publication, and durable Store plus live-workspace fsync if implemented. Preserve operation counts, fixtures, public calls, and timing boundaries.

Prioritize:

• SDK edit families, including canonical-count and capped length-changing cases: short edit + Commit latency.

• payload_create_read, init_namespace, store_footprint, dedup_cross_file, and dedup_cdc_locality: admission transaction count, dirty bytes, and journal/write amplification.

• dedup_workspace_reuse, mixed_load_bearing, and git_tool_workflow: sequential application synchronization.

• dedup_branch_history, especially unrelated history: 200 file fsync calls per step, or 100,000 at tier 500, plus directory syncs and repeated Commit.

• Tiny-file churn, namespace/directory operations, workspace locality, read-only execution, and clean Commit: retain batching and identify unnecessary persistence work.

• Report actual physical flush count/time, SQLite transaction count, Exec/admission/publication latency, p50/p95/p99, throughput, CPU/RSS, and storage growth. Keep preparation and recovery verification outside their unrelated performance timers.

• Account for journal/WAL sidecars in storage metrics and adapt the current closed-Store preparation/clone compatibility checks; do not reuse incompatible prepared Stores or omit durability files.

• Run paired measurements on the supported host-store topology, then regress affected family members and reliability proofs. Do not remove workload fsync calls or weaken the promised acknowledgement boundary to pass a target.

Planning expectation from discussion: roughly 30% more time for ordinary writes and around 2x small-edit latency is acceptable as an initial tradeoff, but these are unmeasured budgeting assumptions, not acceptance thresholds or published performance claims. Per-file durable fsync can make history cases much slower. Set numerical release targets from the measured implementation and an explicit decision on that contract.

Done when

The chosen v0.3.0 durability boundary is implemented, crash/recovery evidence demonstrates the stated guarantee, supported Stores migrate/reopen safely, and SDK/CLI documentation and benchmark reports state the actual durability and performance tradeoff. Remote replication guarantees remain aligned with #52 and distinct from local persistence.

Supporting study

Comprehensive Drive9 hybrid-storage study — full document. The local roadmap source is docs/roadmap/0.3/drive9-hybrid-storage-study.md (pending commit/push), linked from the v0.3 roadmap index. Covers SQL/object tiering, local writeback and replay, acknowledgement profiles, publication/GC, observed durability gaps, LayerFS recommendations, and all 18 benchmark families. Source observations are pinned to Drive9 commit cff6d294b452b54a964de2217c137af303602be9; no new performance or crash-test claims.

Storage amplification and retained history

The attached Drive9 study now records the agent-workload finding: borrow acknowledgement and recovery ordering, but do not adopt ordinary whole-file replacement objects as LayerFS's cloud content layout. Preserve the existing CAS/CDC graph and chunk identities.

Drive9 does have reference-sharing copies and delayed GC. An immutable old blob is not automatically recoverable history: it can be referenced state, an unreferenced object waiting through the cleanup delay, or cleanup backlog. GC cannot reclaim versions intentionally retained by the user. Server-side multipart copy avoids unchanged client traffic but still constructs a complete replacement object.

• For repeated local edits, checkpoints and forks, preserve unchanged canonical payload sharing. Incremental payload growth should follow new unique content, with metadata/layout costs measured separately; do not require a complete replacement file payload for each small edit.
• Measure new unique payload, structural metadata, gross physical bytes written, net retained allocation, uploaded/copied/replicated bytes, request counts, and temporary duplication separately. A small net footprint after deletion must not hide full-file rewrite amplification.
• Bound retry, failed-publication, abandoned-work and eligible-GC backlog; measure before/after GC and compaction without deleting retained history or weakening durability to make results pass.
• Compare equal retained roots, persisted paths and acknowledgement scope. Include metadata-only/no-op cases using absolute byte counts, and unrelated-content controls; CAS cannot eliminate genuinely new data.
• Evaluate remote chunk/request overhead and any pack dead space without changing canonical IDs or making packing mandatory before evidence supports it. Coordinate the selected remote layout with v0.3.0: Cloud native integration and S3-compatible persistence #52.

Illustration only: 1,000 separately published edits to a 1 GiB file, with all versions retained, add 1,000 GiB of complete replacement-object payload. Assuming 64 KiB of new unique chunk content per edit, CAS/CDC adds 62.5 MiB of payload plus metadata/layout overhead. These are hypothetical inputs, not measured results or fixed CDC amplification bounds. Freeze numerical acceptance targets from the measured implementation.

[yifanxuaaa]

Attached study: Drive9 hybrid storage for v0.3

Saved the comprehensive study in the LayerFS working tree at:
docs/roadmap/0.3/drive9-hybrid-storage-study.md

The v0.3 roadmap index links the study and this issue. These documentation changes are local and pending commit/push; the full study is included here so reviewers can read it immediately. Drive9 source references are pinned to the verified clean commit cff6d294b452b54a964de2217c137af303602be9. Existing LayerFS source links in this issue copy are pinned to the inspected repository HEAD where available.

Local Markdown file SHA-256: e1fb26f437d2d9098a99ededd1da94c54c27f28176ca33426654da51d14cd625.

Scope: storage tiers and schema; read/write/multipart publication; tier transitions and GC; local shadow/journal recovery; durability profiles and observed gaps; local-only overlays and packing; partial-edit and append-log tradeoffs; LayerFS adoption decisions; all 18 benchmark families; failure and acceptance checklists. This is a source study, not a benchmark or crash-test result.

Findings update (2026-09-07): Added retained-history versus GC-backlog distinctions, agent-workload whole-file amplification, the illustrative 1,000-edit comparison, CAS/CDC placement constraints, and measurable storage-growth requirements. Sections 8.1, 14.3, and 16.1 contain the detailed update. Figures are hypothetical arithmetic, not measured Drive9 or LayerFS performance.

Drive9 hybrid storage study for LayerFS v0.3

Date: 2026-09-07
Status: Source-based architecture study; recommendations are not an accepted implementation specification.
Primary issue: #69 — Durable Commit and explicit fsync/recovery guarantees.
Related issue: #52 — Cloud native integration and S3-compatible persistence.
Parent: v0.3 roadmap (docs/roadmap/0.3/README.md, local roadmap file; pending commit/push).

1. Purpose, provenance, and evidence limits

This study explains how Drive9 combines SQL, object storage, local execution,
writeback, and recovery, and identifies what is useful for LayerFS's durability
and cloud-storage work. It examines storage placement and acknowledgement
boundaries separately: choosing S3 does not specify when a filesystem operation
may return success.

The inspected repository was /Users/yifanxu/Ephemeral-AI-Lab/drive9, at clean
commit cff6d294b452b54a964de2217c137af303602be9.
The upstream commit was confirmed through GitHub. Drive9 source links below are
pinned to that commit. Local LayerFS references describe the working tree at
the study date and must be rechecked when implementation begins.

Evidence labels used throughout:

• Observed: directly visible in the inspected implementation or schema.
• Intended: described by a design document, comment, or public profile;
  this is not an independently verified crash guarantee.
• Inference: a consequence of the observed structure, not a measurement.
• Recommendation: proposed LayerFS work, subject to v0.3 decisions.

No Drive9 deployment, benchmark, database crash, OS crash, or power-loss test
was run for this study. Existing test source was inspected as a map of intended
coverage, not reported as a passing run. The March design overview is a draft;
its db9/fs9 architecture and approximate latency figures must not override the
current code. Cloud-provider durability, replication settings, availability,
and billing were not independently qualified here. D01

2. Findings that matter most

1. Drive9 has three separate policies: persistent byte placement, the
   acknowledgement boundary, and local-only workspace placement.
2. For TiDB/local deployments, small ordinary files live in SQL and larger
   files live in object storage. The default ordinary-file cutoff is 50,000
   bytes; it also guides the client transfer protocol.
3. FUSE can acknowledge local persistence before remote publication. Its
   shadow files, pending index, and journal make this a recovery design, not
   merely a memory cache.
4. Selected tool state is deliberately local-only. Default coding-agent
   performance does not imply remote persistence of every dependency/cache file.
5. Large-file publication has two systems to coordinate: complete the
   object first, then publish its reference in SQL. Cleanup and recovery bridge
   the failure windows; SQL cannot roll back an external object upload.
6. The ordinary object path uses generated blob keys and multipart replacement.
   It is not LayerFS's content-defined, content-addressed chunk graph.
7. Some acknowledgement paths need further correctness scrutiny. In
   particular, interactive fsync ignores certain journal errors, and remote
   directory fsync is not an upload-drain barrier in the inspected code.
8. LayerFS should borrow explicit contracts and ordering, while retaining its
   canonical object model. Start with durable public Commit; decide durable
   live-workspace fsync and remote replication separately.
9. Agent workloads make storage amplification an architectural constraint.
   Frequent edits, attempts, forks, and checkpoints must not turn unchanged
   large-file content into another full retained payload on every publication.
   Delayed garbage collection cannot reclaim intentionally retained history.

3. Architecture: three policies, not one storage switch

flowchart TD
    App[Application or agent] --> Mount[Drive9 FUSE mount]
    Mount --> Policy{Path policy}
    Policy -->|Remote-managed path| Buffer[Memory buffers and local shadow files]
    Policy -->|Selected tool state| Local[Local filesystem overlay]
    Buffer --> Recovery[Pending metadata and journal]
    Buffer --> Upload[Synchronous upload or background commit queue]
    Upload --> Server[Drive9 API and publication authority]
    Server --> SQL[(SQL metadata and small payloads)]
    Server -->|Presign and finalize| Objects[(S3-compatible objects)]
    Upload -->|Large bytes via presigned URLs| Objects
    Local -->|Explicit pack or portable profile| Pack[Archive uploaded through Drive9]
    Pack --> Server
    Client[CLI or SDK] --> Server
    Client -->|Presigned large transfer| Objects

Loading

The FUSE local execution layer runs with the client/mount. It is not the SQL
authority. A CLI or SDK client can use the remote transfer protocol without
using FUSE's local shadows or journal. Direct object-store mounts are a separate
surface and should not be mistaken for the Drive9 SQL-backed namespace.

Policy	Question answered	Typical choices
Persistent placement	Where are the authoritative bytes after publication?	Inline SQL or object storage
Durability/acknowledgement	What has completed when write/fsync/close returns?	Buffered, locally persisted, remotely published
Workspace placement	Which paths enter the remote-managed filesystem?	Remote source/data versus selected local tool state

These decisions interact but are not interchangeable. Moving bytes from SQL to
S3 does not eliminate publication metadata, and marking a path local-only does
not make it remotely durable. D02, D03, D13, D19

4. Persistent schema and provider selection

4.1 Metadata and payload separation

The current schema separates four concerns, while supporting the legacy
files table during the transition:

Table	Responsibility	Representative fields
file_nodes	Path/directory-entry lookup	Path, parent, name, inode/file identity
inodes	File identity and mutable filesystem metadata	Size, revision, mode, status, timestamps
contents	Payload or payload location	Storage class/ref, inline blob, checksum, encryption information
semantic	Search and enrichment	Text, description, embedding, embedding revision

This separation allows metadata operations to avoid fetching large payload or
embedding fields when their query path does not need them. The File Go model
still exposes a combined view in several APIs; table separation is not a claim
that every query is already minimal. D04, D05

4.2 The ordinary-file threshold

Observed: DefaultInlineThreshold = 50_000, and the backend decision is:

store inline = smallInDB && final_size < inline_threshold

Exactly 50,000 bytes goes to the non-inline side at the default setting. The
threshold is configurable with DRIVE9_INLINE_THRESHOLD and advertised through
server status. This applies to ordinary-file routing; specialized layer,
append-log, or local-overlay paths have their own contracts. D02, D03

Provider family	Current SmallInDB result
local, tidb_zero, tidb_cloud_native	True
Persisted legacy TiDB starter provider	True
db9	False

StorageDB9 = "db9" is also the inline storage-class name. That name must not
be read as proof of a mandatory second database service alongside TiDB. The
older proposal's fs9-based small-file description is not the current TiDB/local
implementation, which writes content_blob. D03, D04

4.3 Placement and transfer negotiation

The threshold deliberately informs both database placement and simple PUT
versus multipart transfer. Otherwise, accepting large simple PUTs while keeping
a low SQL-inline cutoff could turn the server into a RAM-buffered S3 proxy.

FUSE distinguishes a heuristic local threshold from a negotiated protocol
threshold. A missing negotiated value is treated conservatively by upload
paths; it must not blindly assume the historical 50,000-byte cutoff. The remote
storage-class header also matters: file size alone cannot establish whether an
existing object is inline after configuration changes. D02, D06

LayerFS implication: do not adopt a file-size cutoff as the definition of
canonical chunk placement. First decide whether physical units are individual
canonical objects, packed objects, or replicated database pages.

5. Read path

5.1 Remote reads

The backend resolves the canonical path through SQL and builds a read plan:

• Inline class: return the blob, size, revision, and timestamp.
• S3 class: return a presigned GET URL and file metadata.
• Missing or unsupported class: return an error rather than invent content.

For specialized server-side operations, a direct read helper can read S3 bytes
through the server. Therefore the architecture does not guarantee that every
large-file read bypasses the server. D07

5.2 FUSE reads

FUSE combines current local edits/shadows with remote reads and read caches.
The ordinary in-memory read cache defaults to 128 MiB total, a 4 MiB per-file
admission limit, and a 30-second TTL, with revision-aware lookup. These are
read-cache settings, not the SQL-inline threshold. Other read mechanisms include
range reads and disk caching; a cache hit and a cold remote read are different
performance cases. D08

Inference: remote publication does not by itself guarantee that another
mount immediately discards every cached view. Durability, publication, and
cross-client cache coherence need separate evidence. LayerFS should retain
pinned immutable snapshot identities and explicitly define when a mount adopts
a newer head.

6. Small-file write path

For a small ordinary file with inline storage enabled:

1. Validate path, size, and applicable quota constraints.
2. Compute checksum, content type, and applicable text metadata.
3. Begin a tenant SQL transaction.
4. Write content and inode/path state, with revision conditions where supplied.
5. Register required semantic extraction/embedding tasks in the same
   transaction where that path requires them.
6. Commit, then notify background workers and perform any post-commit work.

The server's SQL transaction is the publication point for inline content.
Search completion is not generally the file-write acknowledgement boundary:
some providers use database-generated embedding behavior, while other work is
registered for asynchronous processing. Post-commit quota bookkeeping can also
produce an error after the file mutation has committed; clients need a way to
reason about an uncertain or already-applied operation. D09

BatchWriteCtx prepares eligible whole-file inline writes and processes them
in one tenant transaction. It supports per-item validation/conflict outcomes;
it is not a promise that every submitted item succeeds atomically as a batch.
Larger files stay on their separate upload protocol. D10

Inference: batching can reduce requests and database commit barriers, but
only when the public acknowledgement contract allows the writes to share a
transaction. An application waiting after each strict fsync cannot be made
equivalent by silently delaying its guarantee until a later batch.

7. Large-file upload and publication

7.1 Normal multipart lifecycle

sequenceDiagram
    participant C as Client
    participant S as Drive9 server
    participant O as Object storage
    participant D as SQL authority
    C->>S: Initiate upload with target and revision condition
    S->>D: Record pending upload state
    S-->>C: Presigned multipart plan
    C->>O: Upload parts
    C->>S: Confirm upload
    S->>O: Complete multipart object
    O-->>S: Object complete
    S->>D: Transactionally publish reference and revision
    D-->>S: Transaction committed
    S-->>C: Completion response

Loading

finalizeUpload completes the S3 multipart operation before entering the SQL
transaction that publishes or updates the file reference. Revision checks
protect conditional overwrites. A completed object can exist before any
confirmed path refers to it. D11

The direct backend create/overwrite path can instead receive the complete body
and call PutObject itself before its SQL transaction. The multipart client
path avoids that full server payload transit, but the direct path still exists.
Large shadow-file uploads use streaming readers so the entire payload need not
be materialized in the mount's memory. D09, D15

7.2 Failure windows

Failure point	State that may exist	Required interpretation
Before object upload completes	Pending upload and some parts	No confirmed new file version; abort/resume/reap as appropriate
Object completes, SQL publication fails	Unreferenced complete object	Clean up or retain for recovery; do not claim a successful published file
SQL commits, reply is lost	New file version exists	Retry must reconcile the outcome rather than overwrite blindly
Replacement publishes, old-object cleanup fails	New reference plus obsolete object	Storage leak/backlog is preferable to deleting live content
Local-only fsync succeeds, remote upload has not finished	Local recovery state only	Loss of the local device can still lose those bytes

These are architectural windows, not reproduced failures in this study. Drive9
has cleanup and recovery mechanisms around them, but this review does not prove
every lost-response or ambiguous-commit case. LayerFS #69/#52 must freeze those
outcomes explicitly.

8. Overwrites, tier transitions, and object GC

Ordinary overwrite recomputes the final storage choice. Growing an inline file
can produce a new object reference; shrinking an object-backed file can move
its bytes back inline. For S3 overwrite, a new generated blobs/<id> key keeps
the previous object available until the replacement reference is published.
The test suite includes inline -> S3 -> inline transitions and multiple
replacement-object cleanup cases. Those tests were inspected, not executed.
D09, D12

Confirmed obsolete objects are queued as GC candidates rather than immediately
deleted. The inspected enqueue path sets a seven-day delay. The worker checks
owner activity and exact confirmed-reference reachability; for blobs/ it
conservatively postpones collection while the namespace has a non-deleted fork.
It deletes only after those checks succeed. D16

Tradeoff: delayed and conservative GC protects sharing but retains storage
longer. This study does not prove the collector covers every layer, checkpoint,
in-flight upload, or reference-publication race. LayerFS must define its own
complete root/pin set, including active workspaces, pending candidates,
historical Commits, and remote transfers.

The ordinary blob key is not its SHA-256 content identity. A checksum or a hash
of storage_ref serves verification/indexing purposes and must not be confused
with content-addressed deduplication.

8.1 Immutable replacement is not automatically recoverable history

An immutable object is not modified in place. Immutability does not require
keeping it forever, and leaving an obsolete object in storage does not by itself
provide a history index or a supported way to restore that version.

Old object state	Reason for retention	Can ordinary GC remove it?
Referenced by retained state	Required content sharing or history	No, while the reference remains
Unreferenced, still inside the delay window	Deferred safe cleanup	After the delay and required checks
Unreferenced, collection postponed or failing	Conservative policy or cleanup backlog	Only after the blocking condition is resolved

Drive9 is not devoid of storage controls: its copy path can add a new path to
the same file identity, and its replacement path schedules GC candidates.
Nevertheless, retaining complete replacement objects has a different growth
profile from retaining shared CAS chunks. A seven-day delay is a cleanup policy,
not a deduplication mechanism. Namespace-fork postponement can retain objects
beyond that delay. D16, D26

9. Partial edits and append-heavy files

9.1 Multipart patching

Drive9's ordinary S3 patch protocol creates a replacement multipart object:

• Changed parts are uploaded by the client.
• If necessary, the client downloads original bytes for a changed part and
  merges the local modification.
• Unchanged parts are copied within object storage using UploadPartCopy.
• Confirmation publishes the new object reference with the expected revision.

The default part size is 8 MiB, with adaptive sizing for larger transfers.
Consequently a 4 KiB edit can require uploading its larger containing part;
unchanged-part copies still have service-side request and byte-work costs.
Avoiding unchanged client traffic is not equivalent to avoiding all work
proportional to file size. D17

Ordinary direct append/offset-write helpers can read and rebuild existing
content, so the specialized patch/streaming route matters. Do not assume every
API call is incremental solely because the requested edit is small. D09

9.2 Specialized append-log path

The checkout also contains an S3 Express append-log FUSE path, selected by
operator path patterns and server capability. Pure extensions can use tail
append, while resets/non-tail rewrites use a layout-aware full-body path.
SQLite WAL handling includes generation-reset logic for explicitly configured
files. It is not automatic classification of every log and is not the normal
portable S3 multipart contract. Deployment support and provider guarantees were
not exercised here. D18

LayerFS implication: retain CDC/CAS reuse for canonical files. Consider a
specialized append transport only when a measured workload justifies its
provider dependency and separate recovery semantics; do not add it merely
because another filesystem has one.

10. Local writeback state and ownership

Component	Role	Important identity/bound
Write buffer	Current dirty bytes/parts	Open handle and dirty sequence
Shadow store	Local disk staging and partial I/O	Path, base revision, shadow/content generation
Pending index	Pending file metadata, persisted as .meta	Kind, size, mode, base revision, generation
Journal	Ordered replay records	Sequence, checksum, fsync/commit/unlink markers
Commit queue	Ordered background remote work	Payload base revision, expected revision, generation tokens
Read cache	Reusable remote content	Revision and cache lifetime; not a dirty-data authority

The shadow store includes a default 1 GiB write-cache budget and 10% free-space
guard; actual options and enforcement paths must be recorded in any benchmark.
Pinned generations preserve old readers while newer writes get different
ownership. Streaming a large shadow prevents an unbounded whole-file ReadAll
in that path. D14, D15

The queue distinguishes the revision from which bytes were derived from the
revision submitted as a conditional update. Its generation and durable-watermark
checks address a subtle failure: an old buffered snapshot must not adopt a new
revision and overwrite bytes that another operation already made durable.
Cleanup must likewise remove only the shadow/index generation owned by the
completed upload. D20

LayerFS implication: reuse existing workspace mutation generations and
expected-head checks where possible. Adding asynchronous persistence must not
create a second authority that can publish an older candidate over a newer
acknowledged one.

11. Durability profiles and the actual fsync paths

11.1 Public profile contract

Profile	Ordinary write	File fsync	Close
interactive	Local buffering	Intended local shadow/journal persistence; remote work async	Writeback behavior
fsync	Local buffering	Wait for remote publication	Writeback behavior, with existing strict-path exceptions
close-sync	Local buffering	Wait for remote publication	Wait for remote publication through FUSE Flush
write-sync	Wait for remote publication per write	Usually already synchronized	Usually already synchronized
auto	Local buffering	Mount-time RTT selects interactive or strict	Writeback behavior

auto selects interactive above 10 ms RTT and strict otherwise; probe failure
also selects interactive. The probe is an HTTP request measurement, not a
storage-flush measurement. O_SYNC/O_DSYNC promotes a handle to write-sync.
FUSE Flush is the primary close-error channel because Release has no status
return. D13

These are general remote-managed file policies. Local-only paths, layer mode,
SQLite persistent journal handling, and append-log paths must be examined
separately. In particular, SQLite persistent journal paths can require remote
sync even under interactive mode.

11.2 Intended interactive file-fsync sequence

1. Validate that the snapshot can be safely staged, including untouched bytes.
2. Write/truncate the relevant shadow representation and sync the shadow file.
3. Persist pending metadata and its generation.
4. Append an ordered fsync marker and sync the journal.
5. Enqueue remote work and return.
6. After remote completion, record completion and retire only owned staging.

Pending-index persistence calls atomicWrite: temporary-file write, temporary
file sync, rename, and parent-directory sync. Thus a local fsync path can involve
more than one physical synchronization call even without network I/O. D21

11.3 Observed limitations to carry into a correctness review

Observation	Evidence	Consequence for interpretation
Interactive fsync discards journal.Append and journal.Fsync errors in the inspected branches	Dat9FS.Fsync	A success response alone does not prove every required recovery record was persisted
Shadow staging logs some pending-index persistence errors and continues	stageShadowLocked	Synced payload bytes may lack some intended durable recovery metadata
Remote FsyncDir returns OK without draining remote writes; local-overlay directories call a local sync	Dat9FS.FsyncDir	Directory fsync is not equivalent to LayerFS's current all-dirty workspace fence
Journal compaction uses atomicWrite for .compact, then performs another rename to the live journal name	Journal.Compact	Audit the final rename's durability ordering; syncing the earlier rename does not by itself prove the later one
CRC-framed replay can skip corrupt records	Journal scanner/replay	Detection/salvage behavior is not proof that every acknowledged operation remains reconstructible

These are code observations and verification obligations, not a reproduced
data-loss incident or a full Drive9 audit. The atomic-write helper does sync its
parent directory; it would be inaccurate to say Drive9 omits directory syncing
everywhere. The gap is at particular call/error/rename boundaries. D21, D22

Recommendation: LayerFS should choose explicit acknowledgement profiles
rather than weaken the guarantee automatically when an RTT probe fails or grows
slow. Storage or network conditions may change latency, but should not silently
change what a successful durable Commit means.

12. Recovery and failure scope

Journal records are length-prefixed JSON with CRC32 checksums and ordered
sequence numbers. Replay uses later commit/unlink markers to suppress obsolete
fsync records. Surviving .meta entries contain additional information and are
preferred over reconstruction from the journal alone. Recovery re-enqueues
pending uploads, skips preserved conflicts, and rejects/prunes entries without
usable shadow state rather than fabricating payloads. D22, D23

Failure	Interactive/local acknowledgement	Remote acknowledgement
Mount process dies; local volume survives	Recovery is intended to resume from shadow/index/journal	Remote version should remain available under backend contract
Local VM or disk is destroyed before upload	Locally acknowledged bytes may be lost	Published version does not depend on that client's cache
Remote endpoint is unavailable	Local staging may continue until bounds/backpressure intervene	New remote acknowledgements cannot be assumed to succeed
Newer remote revision appears before queued upload	Must preserve base-revision conflict semantics	Older local payload must not roll back the durable watermark
SQL service/storage or object-store failure	Beyond mount-local recovery	Depends on backend configuration and supported failure domain
Power loss during rename, journal sync, or compaction	Requires ordered persistence and propagated errors	Requires corresponding server/storage guarantees as well

Process-kill recovery, orderly remount, SQL transaction success, and a cloud
replication guarantee are four different claims. No test observed in source
should be described as proving all four without inspecting and executing its
actual failure method.

13. Local-only overlays, packing, and layer terminology

The default coding-agent profile routes selected .git, dependency,
environment, build, and cache paths to a local overlay. Explicit remote rules
can override local matches. This policy can remove substantial filesystem
churn from the remote path, but those bytes are not automatically protected by
ordinary remote-file acknowledgement semantics. D19

Profile	Local routing	Automatic pack selection
coding-agent	Selected developer/tool state	None
portable	Same local pattern family	Entire current overlay
none	No built-in local overlay policy	None

Pack/unpack turns selected local overlay state into an archive uploaded through
Drive9. The default archive namespace is /.drive9/packs/, resolved using the
remote root/profile. Packing makes selected state transferable at that archive
boundary; it is not per-write remote replication or an automatic atomic snapshot
of every simultaneously changing local and remote file. D24

Drive9 also calls its Go fs_layer_* overlay/checkpoint feature LayerFS in its
documents. Here that means Drive9's own layer implementation, not proof of
integration with this Rust LayerFS repository. Its layer entries reuse inline
and object-backed payloads; PutFSLayerObject writes a generated
layers/<layer-id>/<id> object and returns a reference for layer metadata.
The layer feature matrix itself distinguishes supported and incomplete
behaviors, so whole-layer atomicity/checkpoint semantics require their own
review. D25

14. Where performance comes from, and what it costs

Mechanism	Work avoided or amortized	Work that remains / tradeoff
Inline small content	Separate small-object GET/PUT requests	SQL bytes, transactions, replication and semantic work
Presigned large transfer	Server payload transit on normal multipart route	Presign/finalize requests, object upload and SQL publication
Local buffers and shadows	Network wait after every buffered write	Local memory/disk, synchronization and recovery bookkeeping
Background queue	Foreground remote wait	Backlog, failure reporting, conflict handling and resource bounds
Inline batch writes	Repeated requests/transactions	Batch construction and per-item outcomes
Multipart patch	Unchanged bytes crossing the client network	Changed-part amplification and server-side copy requests
Local-only path policy	Remote mutation churn	Weaker persistence scope until an explicit pack/remote override
Immutable replacement objects	Overwriting the old payload before publication	Temporary storage growth and garbage collection
Revision/generation fencing	Stale upload or cleanup races	Additional state and coordination

14.1 A useful cost model

foreground durable latency
  = required local staging and local flush time
  + required sequential network exchanges
  + required remote upload and publication time
  + queue/contention delay at the promised boundary

remote work
  = uploaded bytes + copied bytes + metadata transactions + request count

Not every profile includes every term on its foreground path. Moving a term
to the background does not make it disappear from throughput, resource use, or
recovery exposure. Several local sync calls can share physical writeback, while
one logical synchronization may need multiple ordered flushes. Count actual
system calls and persistence events instead of treating application fsync count
as a measured device-barrier count.

14.2 No defensible direct speed comparison yet

The older overview's approximate db9/S3 latency figures are not current benchmark
results. The earlier LayerFS discussion's roughly 30% ordinary-write budget and
2x small-edit budget are also unmeasured planning assumptions. This study does
not establish that Drive9 is faster/slower than LayerFS or that enabling fsync
will incur those particular multipliers.

For a fair comparison, match persisted paths, bytes, acknowledgement scope,
cache state, transaction boundaries, and storage failure domain. In particular,
exclude comparisons that treat a local-only dependency directory as equivalent
to a remotely published one.

14.3 Agent workloads and whole-file storage amplification

Finding: Drive9's ordinary S3-backed replacement layout is a poor default
for LayerFS's repeated small-edit and retained-history workloads. The SQL plus
object-store separation remains a viable architecture pattern; the whole-file
payload unit should not replace LayerFS's CAS/CDC graph.

An agent can repeatedly edit a large artifact, checkpoint successful and failed
attempts, branch for alternative solutions, and revisit earlier results. The
number of retained publications can therefore be much larger than the number
of final deliverables. This is a workload motivation, not a measured claim about
all agent applications.

For Drive9's ordinary S3 path, a small edit can construct a new complete object.
The direct overwrite path uploads the resulting contents; the multipart patch
path uploads changed parts and copies unchanged parts into the replacement.
Avoiding unchanged bytes on the client connection does not establish storage
sharing between the resulting objects. Budget their logical object payloads
without assuming hidden provider-side deduplication. This statement excludes
SQL-inline updates, reference-sharing copies, coalesced writes that never reach
publication, and explicitly configured append-log behavior. D09,
D17, D18, D26

Illustrative retained-version arithmetic

Assume a 1 GiB file receives 1,000 distinct 4 KiB edits, each edit is separately
published, and the original plus all resulting versions remain retained:

Model	Assumption	Additional content retained after 1,000 edits
Whole-file replacement	Each publication retains another complete 1 GiB object	1,000 GiB, approximately 0.977 TiB; 1,001 GiB including the original
CAS/CDC sharing	Each publication introduces 64 KiB of unique chunk content and shares the rest	62.5 MiB, plus changed metadata and physical overhead; original 1 GiB remains shared

The 64 KiB figure is an illustrative assumption, not a CDC bound, observed
LayerFS result, or promise that a 4 KiB edit always creates that amount of data.
Boundary changes, structural metadata, repeated rewrites, encoding, and storage
layout can alter the actual result. The whole-object figure is logical payload
accounting, not a measurement of provider device allocation or billing. These
figures must not be presented as a measured Drive9/LayerFS benchmark.

The conclusion does not depend on these particular sizes: retaining only new
unique chunks has a fundamentally different cost from retaining unchanged
payload inside every separately addressable replacement object.

GC cannot substitute for sharing

If old versions are unreferenced, prompt and safe GC can reduce steady-state
storage, but the delay window still creates temporary amplification. If old
versions are intentionally retained, deleting them would violate the retention
contract. Deduplication reduces repeated content within retained history; GC
reclaims content that no retained state needs. LayerFS requires both.

Local buffering and local-only caches reduce Drive9's remote work, but neither
is equivalent to retaining all published history with chunk sharing. A fair
comparison must keep the same retained versions and persisted path set.

LayerFS also needs explicit physical-storage accounting

CAS alone does not bound history size or guarantee a compact physical Store.
Account separately for:

1. Unique canonical payload reachable from all retained roots.
2. Reachable structural metadata and history records.
3. Database/index/page or pack-layout overhead.
4. Additional replicas and caches beyond the counted authoritative copy.
5. Live staging, upload attempts, and journal/WAL state.
6. Unreachable objects or dead pack regions awaiting reclamation.

The categories must be defined without counting the same bytes twice. Genuine
new content is unavoidable growth; redundant copies, abandoned attempts,
unbounded metadata, and delayed reclamation are different engineering costs.
The existing LayerFS footprint limitation already acknowledges physical Store
overhead and the complexity of alternative layouts. LayerFS limitations

Required design property for v0.3

For local edits that retain substantial canonical content, incremental payload
storage and transfer should be driven by new unique content, with separately
measured metadata/layout costs. The default publication path must not require
another complete file payload solely because a small region changed.

This is a design and validation requirement, not an unconditional worst-case
complexity claim. CDC, new data, and metadata can still produce size-dependent
work. Preserve every promised root, acknowledgement, and history entry while
measuring and reducing amplification.

The current LayerFS CDC profile is 8 KiB minimum, 16 KiB target, and 32 KiB
maximum. Drive9's 50,000-byte file threshold cannot be copied onto these chunks:
it would classify ordinary chunk payloads as small, defeating the intended
large-file split. Conversely, at an illustrative 16 KiB average, storing a
1 GiB payload as one remote object per chunk means approximately 65,536 payload
objects before metadata. Request/index overhead must be measured too.
LayerFS CDC profile

Physical packs or another batching layout may amortize those requests while
preserving individual canonical IDs. They introduce dead-space, indexing,
compaction, and publication/recovery costs, so packing remains an evidence-led
option rather than a mandatory prerequisite or a change to canonical chunking.

Suitability decision: use Drive9 as a reference for persistence boundaries,
ordering, replay, and failure handling. Do not adopt its ordinary whole-file
replacement layout as LayerFS's cloud content model.

15. Mapping to LayerFS's current architecture

Concern	Observed Drive9 approach	LayerFS position / v0.3 implication
Content identity	Ordinary generated blob refs plus checksums	Preserve canonical CAS/CDC identities
Metadata authority	SQL service, revisions, namespace tables	Preserve conditional publication and v0.2 reconciliation
Small/large split	Whole-file inline threshold	Select physical object/page/pack placement without changing semantic IDs
Local dirty state	Named shadows, persisted metadata, journal	Current anonymous spool backing is insufficient for reboot recovery
Acknowledgement	Buffered/local/remote profiles	Durable Commit first; live fsync and replication separately specified
Batching	Inline SQL batches and queued uploads	Batch admission/transfer within the declared acknowledgement contract
Partial edits	Multipart parts, server-side copying	Keep incremental canonical construction and deduplication
Local-only tools	Path policy plus optional packs	Optional product policy; do not silently exclude benchmark bytes
Cleanup	Delayed candidates and reachability/fork checks	Protect all retained roots, active readers and pending publication dependencies
Retry safety	Base revisions, payload generations, conflict handling	Reuse expected-head/mutation-generation concepts; define lost-reply outcomes

Current LayerFS references:

• Store configuration:
  MEMORY journal, synchronous OFF, EXCLUSIVE locking, WAL rejection.
• Workspace file I/O:
  segments are unlinked after opening; workspace fsync validates/observes
  segment state and completes capture rather than flushing recoverable state.
• Workspace publication:
  object admission, staged candidate, and conditional final publication.
• Admission limits:
  batches bounded by 8,191 objects and just under 4 MiB of admitted bytes.

15.1 Minimal recommendation for #69

1. Freeze the public durable-Commit contract and supported failure domain.
2. Implement a recoverable local Store journal/synchronization configuration,
   migration, and error propagation. Do not treat synchronous=FULL with the
   current MEMORY journal as the completed design.
3. Persist canonical dependencies and required history/head state in the
   correct order before returning a durable success.
4. Prove recovery at every acknowledgement cut and measure existing families.
5. Add durable uncommitted-workspace fsync only with a replayable representation
   for both bytes and metadata. Keep its scope explicit if deferred.

Do not create a second journal merely to duplicate a Store transaction log. A
separate live-workspace recovery record is justified only by the requirement to
recover edits that have not entered the canonical Store.

15.2 Relationship to #52's architecture choice

The roadmap already compares an authoritative SQLite service with replication
against immutable object storage plus transactional metadata. Drive9 informs
the second option's boundaries and costs; it does not settle the choice.

• A database-replication approach must prove recovery/remote acknowledgement of
  the accepted database state and account for page/WAL amplification.
• A canonical-object approach must prove complete-root publication, upload
  idempotency, metadata durability, object reachability, and GC safety.
• Neither option should merge independently written SQLite database files or
  replace automatic clean Branch reconciliation with blind last-writer-wins.

Choose one initial topology from evidence. Both backends are not required.

Any selected topology must preserve reuse across retained versions and be
evaluated for storage growth as well as latency. Repeatedly exporting complete
file versions to object storage is not an acceptable default substitute for
publishing the existing CAS graph. Database-page replication may also amplify
writes despite canonical deduplication; measure that cost rather than assuming
the replication transport inherits the graph's efficiency automatically.

16. Benchmark plan covering all current families

Use the current family entrypoints and benchmark rules.
Retain host-owned SQLite, coordinator, canonical publication, and physical spool
backing for existing benchmark runs; do not introduce container-owned SQLite as
a shortcut. Product changes to topology require explicitly versioned comparison
scope. Preparation, verification, and background-drain timing must remain named.

Measure separately:

• P0: current live-visibility baseline, explicitly non-durable.
• P1: locally durable public Store publication.
• P2: P1 plus durable live-workspace fsync, if implemented.
• P3: explicitly remote-durable publication on the selected v0.3 topology.

These are study labels, not proposed public configuration names. If replication
is asynchronous, publish backlog/drain metrics and never label its local
acknowledgement as P3.

Family	Boundary to preserve	Principal durability sensitivity
edit_length_preserving	One SDK edit + Commit	Fixed publication cost on short latency
edit_length_changing	One SDK edit + Commit	Publication plus actual changed canonical data
edit_canonical_chunk_count	One 64 KiB overwrite + Commit	Admission/publication, not chunk-count label alone
edit_length_changing_capped	One capped-result SDK edit + Commit	Same boundary; full logical result size is not new dirty bytes
payload_create_read	Create has file + root sync; read workload has none	Large data writeback and admission; read execution control
tiny_file_churn	Mutating workloads finish with root sync, no per-file sync loop	Metadata versus bulk payload; preserve batching
directory_construction_traversal	Construction root sync; scans have none	Metadata persistence and read-only controls
namespace_mutation	Move/delete then root sync	Namespace transaction and directory semantics
workspace_change_locality	Batched distributed SDK edits, dense rewrite root sync, clean Commit	Dirty volume versus fixed cost; unnecessary clean publication
mixed_load_bearing	One temporary-file sync per episode + final root sync	Sequential application barriers
git_tool_workflow	Explicit editor-save syncs, Git's own behavior, final root sync	Tool-generated barriers and persisted .git scope
dedup_workspace_reuse	N file syncs + two directory syncs	Live persistence before canonical deduplication
dedup_branch_history	N Commits; unrelated case syncs 200 files each step plus directories	Highest repeated-barrier exposure
dedup_cross_file	Native import	Admitted unique bytes and SQL transaction count
dedup_cdc_locality	Native import	Canonical reuse, request/page amplification
init_namespace	Native initialization	Metadata cardinality and batched admission
store_footprint	Initialization plus edit/fsync/Commit	Include journal/WAL/object staging space
workspace_reliability	Proof-only failure/lifecycle cases	Correctness and recovery, not a throughput score

At tier 500, unrelated history contains 100,000 application file fsync calls
before counting directory syncs. At an assumed additional sequential wait of
1 ms per call, those file barriers alone would add 100 seconds. This is an
arithmetic sensitivity example, not a device measurement. Drive9's remote
FsyncDir behavior cannot replace LayerFS's root fence in this workload and
still be called the same synchronization contract.

Collect p50/p95/p99 across sufficient independent samples; do not label a
single sample as a latency distribution. Record:

• Public call, Exec, admission, staging, publication, and drain durations.
• Application sync count, physical sync calls/time, dirty bytes, and SQL
  transactions; attribute profile-induced work to the correct phase.
• Uploaded/copied bytes, remote request count, RTT, queue age/depth, and retry
  work; distinguish required new content from repeated transfer.
• Host and container CPU/RSS, cache/spool allocation, WAL/journal size,
  retained objects, and eventual GC backlog.
• Crash recovery time, exact recovered root/head, pending suffix, and any
  requested operation whose outcome is uncertain.

Prepared Store copying must preserve the selected durability format and required
sidecars, or use an explicitly safe checkpoint/backup method. The current
closed-Store clone checks reject WAL sidecars; those checks must be deliberately
updated rather than bypassed.

16.1 Storage-amplification acceptance evidence

Use existing SDK edit, canonical-count, reuse, branch-history, locality, and
footprint families as the starting point. Preserve their registered workloads;
the 1,000-edit example above is explanatory arithmetic, not permission to
silently replace a family or launch a new benchmark campaign.

• Record new unique canonical payload bytes, new structural metadata bytes,
  gross physical bytes written, and net retained physical allocation separately.
  Net growth alone can hide a full rewrite followed by deletion.
• Record uploaded bytes, object-store copied bytes, request count, and replication
  bytes separately. Reduced client traffic does not prove reduced remote work.
• At equal retained roots, measure bytes added per edit/Commit and total storage
  across successive checkpoints/forks. Metadata-only and unchanged operations
  must report absolute growth instead of an undefined ratio to zero payload.
• Exercise successful writes, retries/lost replies, failed publications, and
  abandoned work. Verify they do not leave unbounded duplicate payload or
  staging outside the declared retention/recovery policy.
• Measure before GC, after the eligible grace period and successful GC, and
  after any explicit compaction. Report unreachable bytes and dead pack space;
  do not delete retained history to make the footprint pass.
• Pair local-edit workloads with genuinely unrelated-content controls so
  unavoidable unique data is not mislabeled as deduplication failure.
• Freeze numerical growth targets from measured baselines and the required
  retention/acknowledgement contract. Reject a layout that systematically
  materializes full replacement payloads for local edits, without converting
  the illustrative arithmetic into a universal constant-bound promise.

17. Correctness and decision checklist

Requirement	Minimal useful proof	Ownership
No false durable success	Inject payload, metadata, journal and sync failures; success must honor the contract	#69
Crash-safe publication	Fail before/after dependencies, staging, head update and reply; reopen exact accepted state	#69
Durable live edits, if selected	Recover content and namespace/mode/size from surviving persistent state	#69
Directory sync semantics	Prove its documented scope, including rename/create dependencies	#69
Lost replies and retry	Identify already-published work without replaying stale bytes	#69 / #52
Remote complete-root publication	Restore after client loss with no client cache; reject missing dependencies	#52
GC safety	Race deletion with upload, publication, readers, forks and retained history	#52
Resource bounds	Endpoint outage plus writes reaches declared backpressure without losing acknowledged state	#69 / #52
Schema/format migration	Open, migrate, recover and safely copy supported old/new Stores	#69 / #52
Honest performance comparison	Same work, paths, bytes, visibility and durability boundary	Both
Retained-history storage efficiency	Repeated local edits reuse unchanged canonical payload; record new payload, metadata, physical writes, transfer and retained allocation separately	#69 / #52
Bounded reclamation backlog	Failed/retried/abandoned work and eligible dead content reach declared cleanup bounds without deleting retained history	#69 / #52

Questions to resolve before implementation is called complete:

1. Does default Commit mean local-durable or remote-durable in each deployment?
2. What survives client loss, server-process loss, OS crash, and whole-volume loss?
3. Must ordinary workspace fsync recover uncommitted edits, and what is the
   directory fsync scope?
4. Which existing transactions are durability barriers, and can any be safely
   batched without changing candidate/recovery semantics?
5. How is a publication identified after a lost reply or an ambiguous SQL error?
6. What pins every remote dependency until publication/reader/retry completion?
7. Which optional local-only files are outside persistence, and how is that
   visible to applications and benchmark reporting?
8. Which numerical performance and recovery targets are supported by measured
   evidence rather than the discussion's initial budgeting assumptions?

The outcome of this study is an implementation direction: explicit guarantees,
ordered publication, persistent recoverable staging when required, bounded
asynchrony, and retained CAS/CDC semantics. It is not a decision to reproduce
Drive9's complete FUSE state machine or adopt its database/provider stack.

18. Pinned Drive9 source index

Each link refers to inspected commit cff6d294b452b54a964de2217c137af303602be9.
Design documents establish intent; source/tests establish implementation and
intended coverage, not a new runtime result.

• D01 — Historical proposal: design overview.
• D02 — Threshold and ordinary routing: backend defaults, server configuration.
• D03 — Provider decisions: SmallInDB and embedding mode.
• D04 — Data models and compatibility: datastore.
• D05 — Split schema: SQL definitions, content operations.
• D06 — Negotiated protocol cutoff: FUSE threshold and storage class.
• D07 — Read routing: read plan.
• D08 — Read cache: defaults and revision-aware cache.
• D09 — Create/overwrite and transaction ordering: create, overwrite.
• D10 — Inline batching: BatchWriteCtx.
• D11 — Multipart publication: finalizeUpload, client conditional transfer.
• D12 — Tier-transition test coverage: transition tests.
• D13 — Durability profiles: contract, RTT selection and policy.
• D14 — Shadow ownership and bounds: ShadowStore.
• D15 — Streaming shadow upload: remote upload.
• D16 — Garbage collection: candidate enqueue, worker checks.
• D17 — Multipart patching: patch protocol, part sizes.
• D18 — Specialized append log: design and scope, FUSE dispatch.
• D19 — Local-only path policy: classification.
• D20 — Queue and stale-payload fencing: queue state, generation-fence tests.
• D21 — Local persistence and acknowledgement: staging, file fsync, directory fsync, atomic metadata write, pending index.
• D22 — Journal framing, compaction and replay: journal implementation, replay tests.
• D23 — Pending recovery: RecoverPending.
• D24 — Portable local overlay: built-in profiles, pack specification.
• D25 — Drive9's own layer storage: object-backed layer payloads, feature matrix.
• D26 — Reference-sharing copy: CopyFileCtx.