Repository navigation
fix(localfs): make local-store writes atomic and durable - #1865
Open
stalingino wants to merge 3 commits into
Open
stalingino wants to merge 3 commits into
stalingino wants to merge 3 commits into
Conversation
put_object wrote objects straight to their final path with fs::write and upload_file copied staged parquet onto the final path with fs_extra, neither with an fsync. A crash, OOM kill or full disk mid-write left a truncated .stream.json, manifest or parquet file at the real path, which later reads treat as a valid object, and a power loss could drop writes already treated as uploaded. Both now write to a uniquely named .partial sibling, fsync it, rename it over the destination (atomic within a filesystem) and fsync the parent directory on Unix, so readers and crashes only ever see the old or the complete new object, the guarantee object stores give a PUT. The temporary file is removed on failure; its .partial suffix keeps it out of every listing filter. This is the pattern the hot tier already uses for its index.
Contributor
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
- A temporary file left by an interrupted write is named <name>.<id>.partial, which still matches contains-style filters such as ".schema" and "ingestor". get_objects and get_ingestor_meta_file_paths now skip it, so it is never parsed as an object on restart. - Missing parent directories are created by the atomic writer, which fsyncs the parent of every directory it creates; fsyncing only the file's own directory did not persist the entries linking new directories. - The copied file is reopened with write access before sync_all: Windows' FlushFileBuffers needs it.
Syncing only the directories a write created left a gap: a directory created moments earlier by a concurrent writer, or by code that never synced it, was skipped, and could be lost on power loss together with a file this write had already reported as committed. After the rename, every directory from the file's own up to the store root is now fsynced. Syncing a directory with no pending changes is close to free.
Author
|
I have read the CLA Document and I hereby sign the CLA |
nitisht
added a commit
to parseablehq/.github
that referenced
this pull request
Nov 19, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1862.
Description
LocalFS::put_objectwrote to the final path withfs::write, andLocalFS::upload_filecopied staged parquet onto the final path withfs_extra::file::copy, neither with an fsync. A crash, OOM kill or full disk mid-write could leave a truncated.stream.json, manifest or parquet file at the real path, which later reads treat as a valid object. A power loss could drop writes Parseable had already treated as uploaded.Both now go through one helper that:
<name>.<ulid>.partialsibling (concurrent writers to the same key never share a temp file),FlushFileBuffersrequires),A
.partialfile left by a crash still matchescontains-style filters such as.schema, soget_objectsandget_ingestor_meta_file_pathsskip.partialnames before applying their filter.This gives
local-storethe all-or-nothing PUT semantics the object-store backends have, using the pattern the hot tier already uses for its index (hottier/local_state.rs).upload_filestill copies and leaves the staged source in place, because the caller removes it after a successful upload.Not addressed:
.partialfiles left behind by a crash are skipped by listings but not deleted. That can follow as a startup sweep if wanted.Testing
storage::localfs: overwrite leaves the new content and no.partial;upload_filecopies and keeps the source; a failing write keeps the old object and removes the temp file; listings skip a leftover.partialschema/ingestor file; missing directories are created.cargo test --lib storage::localfs: 13 passed.cargo clippy --lib -- -D warningsandcargo fmt --checkare clean. (clippy --testsreports 12 errors that already exist onmain, none in this file.)local-store): ingested 200 events through/api/v1/ingest, waited for the sync to the store, and found the parquet and metadata files underP_FS_DIRwith no.partialfiles left. After a restart, a query still returned all 200 rows.This PR has:


Confidence Score: 5/5
The PR appears safe to merge; no new blocking issues were found.Summary
Local-store writes now replace files atomically and sync them before reporting success. The latest update also syncs directories created by other writers.
Reviews (8) · Last reviewed commit: "fix(localfs): fsync the directory chain ..." · Reviewed by Greptile