Skip to content

fix(localfs): make local-store writes atomic and durable - #1865

Open
stalingino wants to merge 3 commits into
parseablehq:mainfrom
stalingino:fix/localfs-atomic-writes
Open

stalingino wants to merge 3 commits into
parseablehq:mainfrom
stalingino:fix/localfs-atomic-writes

Conversation

@stalingino

@stalingino stalingino commented Nov 19, 2026 •

Copy link
Copy Markdown

Fixes #1862.

Description

LocalFS::put_object wrote to the final path with fs::write, and LocalFS::upload_file copied staged parquet onto the final path with fs_extra::file::copy, neither with an fsync. A crash, OOM kill or full disk mid-write could leave a truncated .stream.json, manifest or parquet file at the real path, which later reads treat as a valid object. A power loss could drop writes Parseable had already treated as uploaded.

Both now go through one helper that:

  1. writes to a uniquely named <name>.<ulid>.partial sibling (concurrent writers to the same key never share a temp file),
  2. fsyncs the file (on a handle with write access, which Windows' FlushFileBuffers requires),
  3. renames it over the destination (atomic within a filesystem),
  4. on Unix, fsyncs every directory from the file's own up to the LocalFS root, so both the rename and any newly created directories survive a power loss, whoever created them (a directory with nothing pending is close to free to sync),
  5. removes the temp file on failure.

A .partial file left by a crash still matches contains-style filters such as .schema, so get_objects and get_ingestor_meta_file_paths skip .partial names before applying their filter.

This gives local-store the all-or-nothing PUT semantics the object-store backends have, using the pattern the hot tier already uses for its index (hottier/local_state.rs). upload_file still copies and leaves the staged source in place, because the caller removes it after a successful upload.

Not addressed: .partial files left behind by a crash are skipped by listings but not deleted. That can follow as a startup sweep if wanted.

Testing

  • New unit tests in storage::localfs: overwrite leaves the new content and no .partial; upload_file copies and keeps the source; a failing write keeps the old object and removes the temp file; listings skip a leftover .partial schema/ingestor file; missing directories are created. cargo test --lib storage::localfs: 13 passed.
  • cargo clippy --lib -- -D warnings and cargo fmt --check are clean. (clippy --tests reports 12 errors that already exist on main, none in this file.)
  • End to end on a v3.2.7 release build carrying this patch (local-store): ingested 200 events through /api/v1/ingest, waited for the sync to the store, and found the parquet and metadata files under P_FS_DIR with no .partial files left. After a restart, a query still returned all 200 rows.

This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors. (No user-facing setting or behaviour change, so there is nothing to document.)

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

Local-store writes now replace files atomically and sync them before reporting success. The latest update also syncs directories created by other writers.

  • LocalFS makes object writes visible as complete files and skips leftover partial files.

Reviews (8) · Last reviewed commit: "fix(localfs): fsync the directory chain ..." · Reviewed by Greptile

put_object wrote objects straight to their final path with fs::write and
upload_file copied staged parquet onto the final path with fs_extra, neither
with an fsync. A crash, OOM kill or full disk mid-write left a truncated
.stream.json, manifest or parquet file at the real path, which later reads
treat as a valid object, and a power loss could drop writes already treated
as uploaded.

Both now write to a uniquely named .partial sibling, fsync it, rename it over
the destination (atomic within a filesystem) and fsync the parent directory
on Unix, so readers and crashes only ever see the old or the complete new
object, the guarantee object stores give a PUT. The temporary file is removed
on failure; its .partial suffix keeps it out of every listing filter. This is
the pattern the hot tier already uses for its index.
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

Comment thread src/storage/localfs.rs Resolved
Comment thread src/storage/localfs.rs Resolved
Comment thread src/storage/localfs.rs Resolved
- A temporary file left by an interrupted write is named <name>.<id>.partial,
  which still matches contains-style filters such as ".schema" and
  "ingestor". get_objects and get_ingestor_meta_file_paths now skip it, so it
  is never parsed as an object on restart.
- Missing parent directories are created by the atomic writer, which fsyncs
  the parent of every directory it creates; fsyncing only the file's own
  directory did not persist the entries linking new directories.
- The copied file is reopened with write access before sync_all: Windows'
  FlushFileBuffers needs it.
Syncing only the directories a write created left a gap: a directory created
moments earlier by a concurrent writer, or by code that never synced it, was
skipped, and could be lost on power loss together with a file this write had
already reported as committed. After the rename, every directory from the
file's own up to the store root is now fsynced. Syncing a directory with no
pending changes is close to free.
@stalingino

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

nitisht added a commit to parseablehq/.github that referenced this pull request Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

local-store writes are not atomic or durable (truncated objects after a crash)

1 participant