fix: one crash-safe owner for durable state writes by snimu · Pull Request #2035 · PrimeIntellect-ai/prime-agent

snimu · 2026-09-04T08:59:17Z

A crash or a hostile Windows filesystem moment could destroy durable state across the agent: an interrupted auth.json write truncated every stored credential (logging the user out everywhere), the auth migration renamed oauth.json away and stripped settings.json before its destination was durable, two first-time settings writers silently discarded each other, a crash-torn session transcript swallowed the next message written after resume, a transiently unreadable session-lease owner was judged stale and destroyed while live, and the kernel bootstrap lock could be stolen mid-reclaim, running two venv rebuilds concurrently.

One owner for durable writes

writeFileAtomicSync (temp file beside the destination + atomic rename; options for fsync, guarded best-effort directory fsync, exact mode, and a before-rename hook for validation/ownership) replaces ten hand-rolled temp+rename blocks and three in-place writes:

settings-manager, telemetry, model-registry, refinement, cron-jobs, rlm-subagent-display, command-recovery-journal (compact), daemon-supervisor (config, worker descriptors, update-restart manifest), daemon-supervisor-ownership, session-manager (full-file rewrite, ownership preserved via the hook)
auth-storage (init, withLock, withLockAsync): previously in place — now atomic
Windows: bounded EPERM/EACCES rename retry (antivirus/indexer holding the destination), and the recovery journal's previously unguarded directory fsync — a hard mid-command throw on Windows — now uses the shared guarded variant
Every migrated site keeps its existing durability level (fsync where it fsynced before, none where it didn't)
Python runtime: HarnessState.save() now writes a temp file and os.replace()s it instead of truncating in place

Point fixes on the same theme

Settings first write: withLock only locked when the file existed; the first-write path now re-reads under the late lock, so racing first writers compose instead of last-writer-wins
Auth migration ordering: auth.json is durable before oauth.json is renamed or settings.json rewritten; source cleanup is best-effort once the destination exists
Session lease (two halves that must land together): the rename-collision guard accepts win32's EPERM/EACCES (a stale lease no longer wedges the session until manual rm), and readLeaseOwner distinguishes "absent" from "unreadable" — a transient read failure fails acquisition instead of reclaiming (destroying) a possibly-live lease
Bootstrap lock: rm-then-mkdir reclaim replaced by the candidate-rename / rename-aside pattern the supervisor launch lock already used, extracted into one shared tryAcquireDirLock
Session-file crash repair at open (repairJsonlDamage): truncates an unparseable tail, re-terminates a parseable one, strips zero-fill runs (recovering the glued record when it parses), one diagnostic line — previously the loader hid the damage and the next append merged into the malformed line and was lost

Validation

Seven pins, each proven failing on the pre-fix code:

auth write failing at the replace boundary leaves previous credentials intact (pre-fix: file replaced in place)
racing first-time settings writers compose (pre-fix: second writer discarded)
failed auth.json migration write leaves oauth.json and settings.json apiKeys recoverable (pre-fix: sources destroyed first)
crash-damaged session file repaired at open: torn tail truncated, zero-filled record recovered, next append on its own line (pre-fix: recovered record lost, append merged)
unreadable lease owner never reclaimed (pre-fix: lease destroyed and stolen)
dir-lock: stale reclaim + acquisition + rival attempts leave exactly one owner; reverting the reclaim to rm-then-mkdir fails the pin
Python: a save interrupted mid-write preserves the previous state file (pre-fix: truncated)

Suites: settings-manager(+bug), auth-storage, telemetry, cron-jobs, refinement, session-lease, migrations, session-manager (all files), rlm-ledger, command-recovery-journal, daemon-runtime-lease, proper-lockfile-compromise, daemon-mode — 380+360 green locally; npm run check green; Python unittest 36/36. (auth-storage has 14 pre-existing local-env failures on the dev machine — identical on clean main — from a real Prime CLI config; unrelated.)

Net src LOC

Total src: +613/−266 (net +347); tests: +795/−11 (net +784).
Mechanism: two new utils (~130 lines incl. option docs) + repairJsonlDamage (~60). Deletion: ten duplicated temp+rename blocks, the in-place writes, the rm-then-mkdir lock, and the inline supervisor lock body.

Fixes the defects reported in discussions #1765, #1431 (residual), #2009 (Python half), #1767, #1764, #1866, #1935 (rename-retry class), #1969 (part 2), #1478.

Linear: RES-1267 https://linear.app/primeintellect/issue/RES-1267

Note

Medium Risk
Touches credential storage, session transcripts, leases, and daemon/bootstrap locking—high impact if wrong, but changes are defensive with broad test coverage.

Overview
Centralizes crash-safe durable writes in writeFileAtomicSync (temp file + rename, optional fsync/dir fsync, exact mode, beforeRename hook, Windows rename retries) and routes auth, settings, telemetry, cron jobs, daemon supervisor state, session rewrites, migrations, and related JSON/JSONL through it—replacing in-place auth.json writes and scattered temp+rename copies.

Adds tryAcquireDirLock for link-based directory locks with inode-safe stale reclaim; kernel bootstrap and daemon supervisor launch use it instead of rm-then-mkdir reclaim.

Session JSONL now runs repairJsonlDamage at open (bounded tail check, truncate/recover torn or zero-filled tails) so the next append is not merged into a poisoned line. Session leases treat unreadable owner.json as possibly live (no reclaim) and accept Windows rename collision codes.

Ordering/race fixes: auth migration persists auth.json before touching oauth.json/settings.json; settings first-write re-reads after acquiring the lock; auth.json init uses exclusive create (wx).

Python HarnessState.save() uses temp + os.replace() like the TS path.

^{Reviewed by Cursor Bugbot for commit f0f02d2. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add crash-safe atomic writes and link-based directory locks for durable state

Introduces shared writeFileAtomicSync and tryAcquireDirLock utilities in atomic-file.ts and dir-lock.ts, then migrates auth storage, settings, session persistence, telemetry, cron state, daemon config, command-recovery journal, and harness state to use them.

writeFileAtomicSync writes to an exclusive sibling temp file, handles short writes, enforces requested file mode, optionally fsyncs, runs a pre-rename validation callback, then publishes via atomic rename with unconditional temp cleanup
tryAcquireDirLock uses link-based publication with owner PID, distinguishes absent vs unreadable owners, reclaims only stale locks whose device/inode match, and renames aside stale entries rather than deleting them
Adds JSONL crash-damage repair in session-manager.ts that detects zero-filled tails, recovers valid records, drops torn lines, and skips repair if the file changed concurrently
Fixes auth migration ordering in migrations.ts so auth.json is written and persisted before oauth.json rename or settings.json cleanup; a failed destination write leaves source credentials intact
Fixes settings first-write race in settings-manager.ts so a racing writer's file is re-read after lock acquisition instead of being overwritten
Risk: session-lease acquireSessionLease now retries on Windows EPERM/EACCES rename failures and treats unreadable owner.json as held rather than stale; check session-lease.ts and bootstrap.ts for callers that previously assumed malformed lock state is reclaimable

^{Macroscope summarized f0f02d2.}

writeFileAtomicSync (temp file + rename, optional fsync, guarded best-effort dir fsync, bounded Windows EPERM/EACCES rename retry) replaces ten hand-rolled temp+rename blocks and the three in-place auth.json writes. Every migrated call site keeps its previous durability level; auth.json gains atomicity it never had - an interrupted write can no longer truncate every stored credential - and the recovery journal's unguarded directory fsync (a hard EPERM mid-command on Windows) now uses the shared guarded variant. RES-1267

FileSettingsStorage.withLock only locked when the file already existed; on the first-write path it now re-reads under the late lock so two racing first writers compose instead of the second silently discarding the first. migrateAuthToAuthJson wrote auth.json LAST, after renaming oauth.json away and stripping settings.json - a crash between destroyed the only credential copies; the destination is now durable first and source cleanup is best-effort. RES-1267

The kernel bootstrap lock reclaimed stale locks with rm-then-mkdir: two racers could both judge the same lock stale and racer B's rm could delete the lock racer A had just acquired, yielding concurrent venv rebuilds. The supervisor launch lock already used the correct candidate-rename/rename-aside pattern; it is now extracted into tryAcquireDirLock and both sites use it, with win32's EPERM/EACCES rename-onto-directory errnos treated as collisions at both. RES-1267

A crash leaves exactly two artifact shapes in an append-only JSONL transcript: a torn tail and zero-filled bytes (size committed, page data lost). The loader silently skipped both while the appender wrote the next entry onto the same physical line - the new entry vanished and the damage self-perpetuated across crash/resume cycles. repairJsonlDamage now runs once at open: it truncates an unparseable tail, re-terminates a parseable one, strips zero-fill runs (recovering the glued record when it parses), and reports one diagnostic line. Session rewrites also go through the shared atomic writer, preserving ownership metadata via its beforeRename hook. RES-1267

acquireSessionLease tolerated only EEXIST/ENOTEMPTY on the rename collision; win32 raises EPERM/EACCES for rename-onto-directory, so any existing lease - including a stale one - threw raw instead of reaching the reclaim, wedging the session until a manual rm. Fixing that alone would have activated the second bug: readLeaseOwner collapsed every read error to undefined and undefined meant stale, so a transient EPERM/EBUSY read would have destroyed a LIVE lease. The two now land together: the collision guard accepts the win32 errnos, and an unreadable owner fails acquisition instead of reclaiming. RES-1267

HarnessState.save() truncated the state file in place; a concurrent reader catching the window saw a partial file, degraded to empty state, and could later persist that emptiness over durable knowledge. The save now writes a sibling temp file and os.replace()s it. RES-1267

Short writes loop until every byte lands (a partial temp is never renamed in); auth.json initialization uses an exclusive create so a racing initializer cannot replace credentials another process saved between the absence check and the write; the migration's auth.json write fsyncs file and directory - the destination-first ordering guarantee is otherwise hollow on journaled filesystems; dir-lock reclaim re-checks the moved lock's identity and puts back a lock that changed owners mid-reclaim; session repair resolves symlink aliases and restores ownership exactly like the rewrite path; the Python save uses a unique temp name and preserves the destination mode; the repair helper is no longer exported. RES-1267

RES-1267

The settings rewrite in the auth migration now preserves the file's own mode instead of recreating it umask-open (a deliberately relaxed mode also survives - nothing is silently tightened). The dir-lock owner parse only accepts positive pids: kill(0)/kill(-n) probe the caller's own process group, so a zero or negative pid in a lock read as held-forever and wedged bootstrap. The Python save creates its temp file with the destination's mode (0600 fallback) at open time, closing the umask-wide window while json.dump streams secrets. And the session repair aborts its rename when the file changed size or mtime since the snapshot - a concurrent writer's appends always win over a stale repair; the next open retries. RES-1267

parseInt accepts "123garbage" as 123, so corrupted pid-file content whose numeric prefix matched a live process read as held forever. Only an exact positive integer now names an owner; anything else is judged by the caller's staleness rule, matching garbage-is-reclaimable semantics. RES-1267

RES-1267

…e-owner

…parity The lock is now a hard-link-published FILE: the owner pid is written to a private temp and published with linkSync, so the lock is born with its content and EEXIST is the only collision signal - the EEXIST/ENOTEMPTY/EPERM/EACCES rename-interpretation matrix is gone, and an NFS-style failure report after a landed link is recovered by the classic nlink==2 verification. Reclaim keeps the identity check; a swapped-in live FILE lock is put back with a link (which cannot clobber a third acquirer). A directory at the lock path is judged as a legacy lock from the replaced protocol for one release. Session repair no longer scans every transcript at open: a bounded tail read (NUL bytes, missing trailing newline, malformed final line) gates the full scan, so clean opens of large sessions stay O(window). Atomic writers now resolve symlink aliases like the in-place writes they replaced: auth.json and harness_state.json (TS and Python) write through an alias to the real file instead of replacing the link. RES-1267

An unreadable owner (transient EPERM/EBUSY) now reports the lock held instead of judging it stale and destroying a live lock - the same absent-versus-unreadable split session leases got. A candidate-cleanup failure in the finally can no longer mask a settled acquisition as an error. And the put-back after an identity mismatch branches on the moved entry's actual file-vs-directory type instead of the judgment snapshot: linking a directory fails, which silently discarded a rival's legacy lock swapped in mid-reclaim. RES-1267

A cleanup-blocked or crashed acquirer leaks its uniquely-named candidate file; retry loops could accumulate them forever. Each acquire now does one bounded, best-effort pass over the lock's directory removing candidates older than an hour - the prefix can never match the lock itself and the age gate spares every mid-publish rival. RES-1267

RES-1267

The comment dedupe left empty catch blocks that the no-silent-catch guard rightly rejects; the guard outranks the trim. And an acquirer suspended past the sweep age (machine sleep) loses its candidate to a rival's litter collection between the temp write and the publish link: the ENOENT now triggers exactly one retry with a fresh candidate instead of surfacing as a raw stat failure. RES-1267

A transient stat on the moved-aside entry collapsed to "not a directory", sending a legacy dir down the link branch whose failure deleted a live lock. Probes now answer file | directory | absent | unknown, and unknown is never destructible: an unjudgeable lock reads held, an unjudgeable moved entry is restored (rename-back) or left aside - never deleted - and the publish recheck only treats a definite ENOENT as a swept candidate instead of inventing that answer from any probe failure. RES-1267

The migration's settings.json rewrite now resolves the alias like the auth and session writers do, so a dotfiles-managed settings symlink survives and the legacy apiKeys leave the REAL file. And realpathIfPresentSync now walks a dangling symlink chain to its (absent) target - in-place writes created the target through the alias, so the exclusive initializer and atomic replace must too, instead of renaming over the link. RES-1267

The judged lock is captured once as dev+ino (+shape) and every later decision keys off that capture: the aside entry is verified by inode (not re-probed content or shape), a mismatch restores by the captured shape and NEVER deletes - any restore failure leaves the entry aside - and a rename-out whose reply was lost (NFS EIO after completion) is recognized by re-probing the path and the aside's inode instead of surfacing a raw error. Zero-inode filesystems fall back to held. RES-1267

The inode-identity pin now covers pid reuse (same owner content, new inode: restored, never deleted), and the dangling-symlink walk resolves relative targets against the link's PHYSICAL parent directory, matching how the kernel resolves them for in-place writes through a symlinked directory. RES-1267

Two consistency amendments to the inode-identity reclaim: the restore operation is chosen by the ASIDE's actual shape (already known from the verify stat) instead of the judgment capture, so a file/dir swap is restored at the path rather than deterministically failing; and a lost-reply rename with a mismatched aside falls through to the normal verify - something WAS moved - restoring the swapped rival instead of reporting reclaimed and stranding it. RES-1267

Two closures on the lock reclaim. An unreadable-shape restore now attempts linkSync only - it can never replace a rival's freshly published lock (EEXIST) and fails harmlessly on a directory (EPERM), leaving the entry parked aside either way; rename-back stays reserved for a known directory. And the judged inode is pinned with an open descriptor for the whole reclaim: Linux recycles inode numbers immediately, so a rival's fresh lock could reuse the judged inode and read as identical - with the descriptor held, the number cannot be reassigned and the dev+ino identity is sound (verified 15/15 on Linux where the unpinned version failed 14/15). RES-1267

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7637cc5. Configure here.}

First-run auth.json creation fchmods the descriptor to exact 0o600 - a restrictive umask (0o700) masked the open mode to 0o000 and the very next locked read failed EACCES. A benign trailing blank line no longer reads as damage, which re-ran the full repair scan on every open. The auth migration resolves a dangling auth.json symlink to its configured target instead of replacing the link. The dangling-chain walk fails loudly after 32 hops rather than silently renaming over an intermediate link (or looping on a cycle). And the inode pin descriptor closes on every exit path, including the pin-mismatch return. RES-1267

Same-shape pins table-drive (legacy pid hygiene, symlinked-auth variants, one-full-read gate cases), the two lost-reply pins merge into one two-phase test, the atomic-write pair shares one destination, and repeated scaffolding moves into seedDirLock/makeAgentDir builders. Multi-line comments collapse to one-line invariant guards. No behavior coverage changes. RES-1267

snimu added 6 commits September 4, 2026 10:57

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

cursor Bot reviewed Sep 4, 2026

View reviewed changes

snimu added 2 commits September 4, 2026 11:17

test: pin symlink-alias repair with preserved mode

c734fd3

RES-1267

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/migrations.ts Outdated

Comment thread packages/coding-agent/src/core/kernel/bootstrap.ts

Comment thread prime-agent-runtime/src/rlm/harness.py Outdated

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/core/session-manager.ts Outdated

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

docs: state the fully-formed lock-publish invariant

4184a03

RES-1267

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

snimu added 2 commits September 4, 2026 12:52

style(coding-agent): fix test indentation

3278cd3

Merge remote-tracking branch 'origin/main' into fix/atomic-persistenc…

79ba09b

…e-owner

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/core/session-manager.ts

Comment thread prime-agent-runtime/src/rlm/harness.py Outdated

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/core/auth-storage.ts Outdated

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

Comment thread packages/coding-agent/src/utils/dir-lock.ts

Comment thread packages/coding-agent/src/utils/dir-lock.ts

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

snimu added 2 commits September 4, 2026 15:19

chore: dedupe two literal comment pairs

b4ac0f2

RES-1267

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/migrations.ts Outdated

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/atomic-file.ts Outdated

snimu added 2 commits September 4, 2026 16:18

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

snimu force-pushed the fix/atomic-persistence-owner branch from 2e8266f to 7637cc5 Compare September 4, 2026 16:28

macroscopeapp Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/core/session-manager.ts

Comment thread packages/coding-agent/src/migrations.ts Outdated

Comment thread packages/coding-agent/src/core/auth-storage.ts Outdated

Comment thread packages/coding-agent/src/utils/atomic-file.ts Outdated

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/coding-agent/src/utils/dir-lock.ts Outdated

snimu added 2 commits September 4, 2026 18:50

Uh oh!

Conversation

snimu commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

One owner for durable writes

Point fixes on the same theme

Validation

Net src LOC

Add crash-safe atomic writes and link-based directory locks for durable state

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

snimu commented Sep 4, 2026 •

edited

Loading