Skip to content

local-store writes are not atomic or durable (truncated objects after a crash) #1862

Open

Description

@stalingino

/about output: version v3.2.7 (also reproduced on main at 0de90cf), mode local-store.

Description:

With local-store, objects are written non-atomically. LocalFS::put_object writes straight to the final path with fs::write, and LocalFS::upload_file copies the staged parquet onto the final path with fs_extra::file::copy. Neither fsyncs.

Object stores give a PUT all-or-nothing semantics: a reader sees the old object or the complete new one. A plain write to the final path does not. A crash, an OOM kill or a full disk mid-write leaves a truncated .stream.json, manifest or parquet file at the real path, and later reads treat it as a valid object. Without an fsync, a power loss can lose writes Parseable already treated as uploaded and removed from staging.

Expected: local-store writes have the same all-or-nothing guarantee as the object-store backends: write to a temporary sibling, fsync, rename over the destination, fsync the directory. The hot tier already does this for its index (hottier/local_state.rs).

Steps to reproduce: run local-store, ingest continuously, and kill -9 the process (or fill the disk) during a sync. Partially written files can then appear under P_FS_DIR.

Activity

  1. nikhilsinhaparseable commented on Nov 19, 2026

    @nikhilsinhaparseable
    Member

    @stalingino do you see a corrupt file in data or staging directory after SIGKILL? what is the behaviour when you restart the server?
    In case of staging, the conversion task deletes the corrupt file and reconstructs the parquet from arrow files present in staging directory.
    In case of data directory, corrupt file will make the dataset irrecoverable and query on the dataset will always fail.
    please let us know the behaviour you see.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions