feat(vfs): sparse writes v3, session-scoped coverage staging - #187
Open
XciD wants to merge 2 commits into
Open
Conversation
Open-for-write on a CAS-backed file punches a sparse hole instead of downloading the content; reads fill holes lazily from CAS and cache them into staging; flush composes the new revision via range_upload from the dirty ranges only. Off by default behind --sparse-writes (implies --advanced-writes), with a small-file threshold (default 256 MiB, HF_MOUNT_SPARSE_MIN_BYTES) below which the standard download-then-upload path stays in effect. Third iteration of this feature. The previous two (#41, #180) kept accumulating corruption edge cases rooted in the same structural causes; this one removes the causes instead of patching their symptoms: - Session-scoped sparse state: SparseWriteState lives strictly from the first write-open to the last clean close (release-side drop plus commit-apply drop when no handle remains). No retained cross-session state for poll rotation, reopen reclassification, or setattr drift rebuilds to interact with. - Identity freeze: update_remote_file refuses to rotate xet_hash/size while ANY handle is open (callers only invalidate kernel caches when the rotation applied), and open's drift check covers both the sparse and the download path, so an open racing a remote update retries instead of committing old bytes under a new identity. - Untrusted staging discipline: a staging file is reused only when the inode marks it meaningful (is_dirty || staging_is_current); a bare exists() is never trusted, killing the leftover-file-committed-as- content class. Decisions are re-read under the install write lock, not from pre-lock snapshots. - One flush route per item: UploadRoute (FullStaging | SparseCompose) is computed once and drives Pass A, Pass B membership, and the commit dispatch; routes cannot disagree. - Bounded everything: per-round snapshot budget (256 MiB default) with torn-compose abort on concurrent writes, dirty-range coalescing over covered gaps above 4096 entries, CAS-fetch retry parity with the lazy read path, and flush retry carryover with exponential backoff and a clamped floor. - No panics under locks: invariant violations surface as EIO with the entry unmutated (and this call's staging creation undone).
Carries the accumulated test corpus from both previous attempts plus regression tests for every corruption class found in review: - VFS invariant tests: open skips download, lazy hole fill and staging cache, flush composition, setattr paths, drift handling, no-op flush hash retention. - Regression tests: reopen-after-fast-path-commit write loss, rotation under open handle (chimeric size/content commit), torn multi-round compose, untrusted staging leftovers, non-Xet rotation EIO (lock poisoning), session-scoped state drop, flush retry convergence after Pass A/Pass B failures, snapshot budget bounding, eligibility-predicate unification, transient CAS failure retry on sparse reads. - Same-inode multi-worker stress harnesses (mock CAS) with byte-level oracle verification. - fsx_paranoid (CAS round-trip per mutation) and sparse_concurrent_real (multi-worker same-inode against real CAS) wired into the fsx CI job with --sparse-writes.
Contributor
POSIX Compliance (pjdfstest) |
Contributor
Benchmark Results |
XciD
marked this pull request as ready for review
June 13, 2026 08:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Third attempt at the sparse-write feature, replacing #41 (
feat/append-writelineage) and #180 (coverage-map rewrite). Both previous attempts converged on working happy paths and then accumulated corruption edge cases in review. Two deep review passes over #180 produced an inventory of every failure found across both attempts; this PR was rebuilt frommainaround the structural causes instead of patching symptoms.The requirement, re-analyzed
Edit huge CAS-backed files in place (the delta-weight-sync workload: open an existing model file, write small deltas, fsync, close) without downloading the file at open and without re-uploading it whole at flush. NFS and FUSE. Multi-client is last-writer-wins. Off by default behind
--sparse-writes(implies--advanced-writes); files under a threshold (default 256 MiB,HF_MOUNT_SPARSE_MIN_BYTES) keep the standard download-then-upload path, where CDC dedup already makes small files cheap.Why the previous attempts kept breaking
Every corruption bug found in #41 and #180 traces to one of five structural causes:
xet_hash/sizewhile handles were open or an open was mid-download, so flushes committed old bytes under new identities, with no self-healing.staging_is_current,sparse_write, per-handle flags andis_dirtydesynced; decisions made from pre-lock snapshots were stale in exactly the dangerous direction.Design principles in this PR (each kills a class)
SparseWriteStateis session-scoped: dropped at last clean close (release side) and by commit-apply when no handle remains. Idle inodes carry no sparse state at all.update_remote_filerefuses to rotate while any handle is open (the deletion path already did this); callers only invalidate kernel caches when a rotation actually applied; open's drift check covers both the sparse and the full-download path (EAGAIN retry).UploadRoute), computed once, drives Pass A, Pass B membership, and commit dispatch. Failed/aborted commits return to the loop and retry with exponential backoff (clamped floor, no busy-mount collapse, abandoned at shutdown per the #186 bounded-drain philosophy). Missing-staging inodes resolve to a terminal state instead of blocking rotation forever.What is carried over
The coverage-map core of #180 (
coverage/dirty_rangesinvariant,dirty_ranges ⊆ coveragestructural), therange_uploadxet-core integration, and the full test corpus of both PRs.Tests
--features fuse,nfs(407 default), including a regression test for every corruption class found in the two review passes, each proven red against the code it fixes.fsx_paranoid(CAS round-trip after every random mutation) andsparse_concurrent_real(multi-worker, real CAS) wired into the fsx CI job with--sparse-writes.fmt --check, clippy-D warningson all four feature combos, lib tests withfuse,nfs,nfs_ops4/4.Known limitations (documented in code)
Supersedes #41 and #180.