Skip to content

fix(daemon): allow supervised restarts to become ready - #23

Merged
shitratgit[bot] merged 2 commits into
mainfrom
fix/restart-readiness-window
Sep 16, 2026
Merged

shitratgit[bot] merged 2 commits into
mainfrom
fix/restart-readiness-window

Conversation

@shitratgit

@shitratgit shitratgit Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Why

A live supervised restart on Flagg took about 22 seconds to become healthy. The CLI stopped waiting at 15 seconds and reported failure even though launchd had accepted the restart and brought up a healthy replacement.

That false failure sends agents toward break-glass recovery when routine headless restart already worked.

What changed

  • extend the post-acknowledgement readiness window from 15 to 30 seconds
  • name the timeout once and use it in the failure message

Verification

  • go test ./cmd/secrets ./internal/daemon
  • live RPC restart replaced PID 349 with PID 98339
  • fresh daemon returned 258 secrets, 49 active leases, and uptime: 0m

The separate joelclaw host-bootstrap PR installs exact passwordless launchd recovery for a genuinely wedged RPC loop.

Intent:
- Stop reporting a failed daemon restart when launchd has accepted the exit and the replacement needs more than 15 seconds to answer.
- Preserve headless recovery without sending operators toward unnecessary break-glass UI.

Implementation:
- Name the readiness deadline as daemonRestartTimeout.
- Extend the post-acknowledgement readiness window from 15 to 30 seconds.
- Derive the failure text from the same constant.

Outcomes:
- The CLI covers the 22-second replacement observed on Flagg instead of emitting a false failure.

Verification:
- go test ./cmd/secrets ./internal/daemon passed.
- A live RPC restart replaced PID 349 with PID 98339; status then reported 258 secrets and uptime 0m.

Follow-ups:
- Host bootstrap still needs the separate joelclaw sudoers repair for genuinely unresponsive RPC.
@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 554e4ac6-af49-400f-8d1d-95c111739729

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Intent:
- Stop healthy launchd processes from becoming intermittently unresponsive during the once-per-minute expired-lease sweep.
- Keep status, list, lease, and restart RPCs responsive when audit persistence is slow.

Implementation:
- Remove and copy expired leases while holding the lease mutex.
- Release the mutex before writing one durable audit entry per expiration and persisting the compacted lease set.
- Add an injectable audit function and a concurrency regression that blocks audit logging while proving List remains responsive.

Outcomes:
- Audit fsync latency no longer serializes every RPC that reads the lease table.

Verification:
- go test -race ./internal/lease ./internal/daemon ./cmd/secrets passed.
- Regression proves List returns while expiration audit logging is deliberately blocked.

Follow-ups:
- Deploy the rebuilt daemon through the root-owned service installer after merge; the running service binary is intentionally not replaced through an unprivileged path.
@shitratgit

shitratgit Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Added the underlying contention fix in 8ba568a.

The daemon was not crashing. CleanupExpired held the global lease mutex while emitting and fsyncing one audit record per expired lease. The minute sweep therefore made launchd look healthy while status, list, lease, ShitRat auth, and restart calls queued or timed out.

The fix now removes/copies expired leases under the mutex, releases it, then performs audit and persistence work. A race-enabled regression blocks audit logging and proves List remains responsive.

Verification: go test -race ./internal/lease ./internal/daemon ./cmd/secrets passed.

@shitratgit
shitratgit Bot merged commit 295a940 into main Sep 16, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants