You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
local-store writes are not atomic or durable (truncated objects after a crash) #1862
/about output: version v3.2.7 (also reproduced on main at 0de90cf), mode local-store.
Description:
With local-store, objects are written non-atomically. LocalFS::put_object writes straight to the final path with fs::write, and LocalFS::upload_file copies the staged parquet onto the final path with fs_extra::file::copy. Neither fsyncs.
Object stores give a PUT all-or-nothing semantics: a reader sees the old object or the complete new one. A plain write to the final path does not. A crash, an OOM kill or a full disk mid-write leaves a truncated .stream.json, manifest or parquet file at the real path, and later reads treat it as a valid object. Without an fsync, a power loss can lose writes Parseable already treated as uploaded and removed from staging.
Expected: local-store writes have the same all-or-nothing guarantee as the object-store backends: write to a temporary sibling, fsync, rename over the destination, fsync the directory. The hot tier already does this for its index (hottier/local_state.rs).
Steps to reproduce: run local-store, ingest continuously, and kill -9 the process (or fill the disk) during a sync. Partially written files can then appear under P_FS_DIR.
@stalingino do you see a corrupt file in data or staging directory after SIGKILL? what is the behaviour when you restart the server?
In case of staging, the conversion task deletes the corrupt file and reconstructs the parquet from arrow files present in staging directory.
In case of data directory, corrupt file will make the dataset irrecoverable and query on the dataset will always fail.
please let us know the behaviour you see.
Activity
nikhilsinhaparseable commentedon Nov 19, 2026
@stalingino do you see a corrupt file in data or staging directory after SIGKILL? what is the behaviour when you restart the server?
In case of staging, the conversion task deletes the corrupt file and reconstructs the parquet from arrow files present in staging directory.
In case of data directory, corrupt file will make the dataset irrecoverable and query on the dataset will always fail.
please let us know the behaviour you see.