Skip to content

[codex] Use atomic write in _update_upload_status to prevent file corruption - #350

Open
rahulrao85 wants to merge 1 commit into
huggingface:mainfrom
rahulrao85:fix/session-uploader-atomic-write
Open

[codex] Use atomic write in _update_upload_status to prevent file corruption#350
rahulrao85 wants to merge 1 commit into
huggingface:mainfrom
rahulrao85:fix/session-uploader-atomic-write

Conversation

@rahulrao85

Copy link
Copy Markdown

Summary

Replace the seek(0) → json.dump → truncate() in-place rewrite pattern in _update_upload_status() with a temp-file + os.replace() atomic write pattern.

Problem

The previous implementation used:

f.seek(0)
json.dump(data, f, indent=2)
f.truncate()

If json.dump() raised an exception mid-write (e.g. serialization error for a non-serializable value), the file was left with partially-overwritten data at a wrong truncation position — corrupting the session JSON file permanently. Both the org and personal uploaders use this same function, so a crash in either process would corrupt the shared session file.

Fix

  1. Read the session file under the exclusive lock (unchanged)
  2. Write the full JSON to a .tmp sibling file using json.dump
  3. Atomically swap with os.replace(tmp_path, session_file)

If json.dump crashes, the .tmp file is cleaned up and the original session_file remains untouched. os.replace() is atomic on POSIX and modern Windows (NTFS atomic rename), so readers always see a consistent file.

Validation

  • Same locking semantics preserved (exclusive lock protects the read)
  • os.replace is used instead of shutil.move for cross-filesystem safety (ensures atomic rename when source and dest are on the same filesystem)
  • Temp file cleanup in except block prevents orphaned .tmp files

…file corruption

The _update_upload_status function used seek(0) -> json.dump -> truncate()
to rewrite the session JSON file in-place. If json.dump raised an exception
mid-write (e.g. serialization error), the file was left with a corrupted mix
of old and new data at a wrong truncation point.

Replace with a temp-file + os.replace pattern: the full JSON is written to a
.tmp sibling file first, then atomically swapped into place. On failure the
.tmp file is cleaned up and the original file remains untouched.

Signed-off-by: Rahul Rao <rahulrao85@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant