Skip to content

fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error - #2047

Open
snimu wants to merge 7 commits into
mainfrom
sebastian/worker-state-recovery
Open

fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error#2047
snimu wants to merge 7 commits into
mainfrom
sebastian/worker-state-recovery

Conversation

@snimu

@snimu snimu commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Part of the worker-state single-truth program (Linear RES-1270); squashes the confirmed residuals of discussions #1742 and #1650 and the mechanism behind #1870/#1641 (and the #1904d reconnect gap).

Purpose

Two owners for two worker-state facts:

  1. Liveness/failure. After MAX_DEFERRED_RECOVERY_ROUNDS the recovery ladder parks a live-but-silent worker lifecycle='failed' terminally: no timer re-probes it, reuseWorkerForCreate hard-rejects it, and only client-owned workers auto-recover on attach. A worker frozen longer than the ~2-3 min probe budget (dark wake, paused VM, swap-in) stays written off even though isProcessAlive + processStartId prove it is ours and alive ([Bug] Sleeping the machine permanently orphans daemon sessions: a live worker is marked failed and never re-checked #1742 residual). The create path threw exactly in the gap where neither retry_worker nor a launchEnv-bearing fresh create applied ([Bug] Prime Agent 0.8.0: saved session becomes permanently unopenable after failed worker recovery #1650 residual).
  2. Addressability. findWorker resolves selectors only through the in-memory roster, hydrated lazily from live workers after adoption; the persisted worker descriptors that carry rootActiveSessionId/rootSessionId were consulted only in the client-owned attach special case. Every supervisor replacement re-opens a window where attach/prompt/kill on a provably-known session answer Unknown active session, and clients cannot distinguish gone from recovering (v0.8.1: supervisor restart loses live worker active-session mapping during RLM fan-out #1870, [Bug] v0.8.0 supervisor restarts make every TUI fail to reattach with “Unknown active session” #1641).

Change

  • canRetryFailedWorker (failed + no stop in flight + process identity verified current) and retryWorkerRecovery (the state reset retry_worker already performed) become the single retry semantics. retry_worker, attach, and create reuse all share them; the parked state is now exit-able on any touch. Identity gone/replaced/unverifiable keeps the old hard answers.
  • The client-owned attach pre-pass generalizes to any descriptor-matched worker (owned keeps its launchEnv/recoveryConfig relaunch semantics; non-owned recovers only when identity-current-failed, and never blocks on an in-flight recovery). A child-session attach onto a failed worker retries after roster resolution.
  • findWorker on a roster miss consults descriptors: a root session on a recovery-candidate worker answers with a structured, retryable session_recovering error (DaemonSessionRecoveringError, new DaemonErrorInfo variant, DAEMON_SCHEMA_REVISION 27). Failed workers deliberately keep Unknown active session so first-party clients take the saved-session/create fallback, which reclaims or retries them (that fallback is what self-heals identity-gone workers via launchEnv).
  • main.ts (get_state lookup) and the agents view treat session_recovering like the unknown fallback, converging on the create path.

Wire classification: backward-compatible. The new variant rides the existing optional errorInfo; old clients see a readable message, old daemons never send it, nothing is capability-gated because it degrades to a plain error.

Net src LOC: +102/-38, of which the supervisor is +84/-38 (about half is the owned-attach special case becoming the general pre-pass, plus the two shared helpers replacing three inline copies of the reset).

Tests

  • New pin: create reuse on a failed worker with a current process identity retries recovery instead of throwing (fail-unfixed verified: forcing the old terminal semantics rejects with "could not be safely reclaimed").
  • New pin: a descriptor-known unaddressable root session answers session_recovering (typed, with activeSessionId); a failed worker and a truly unknown selector keep Unknown active session (fail-unfixed verified: removing the fallback yields Unknown).
  • Both-direction wire pins in daemon-errors.test.ts: serialize/deserialize round-trip for new clients, readable message for old clients, and plain old-daemon failures do not misclassify.
  • Existing pins kept green: retry-while-stopping rejection, intentional-stop tombstone reset, failed-unreclaimable create rejection (identity gone), owned-session recovery paths.

Ran locally: daemon-supervisor-{monitor,process,eviction,admission,input-pause}, daemon-agent-roster, daemon-supervisor-lazy-subagents, daemon-errors, regressions 4656 + 4603, agent-connection-daemon, daemon-ps, agents-view suite, main-interactive-routing — 540 tests pass; npm run check green.


Note

Medium Risk
Touches core daemon supervisor routing, attach/create, and worker lifecycle; behavior changes for failed-but-live workers and session lookup errors, though guarded by process identity and extensive new tests.

Overview
Fixes daemon sessions getting stuck when a worker is parked failed but its process is still alive, and when the supervisor knows a session id but cannot route to it yet.

Failed worker recovery: Shared canRetryFailedWorker / retryWorkerRecovery replace duplicated reset logic. retry_worker, session create reuse, attach, and any command forwarded to the worker now automatically retry recovery when failure is not terminal (current process identity, no stop in flight). User-stopped workers stay stopped until an explicit retry_worker.

Addressability: When roster lookup misses but a persisted descriptor matches a recovering (non-failed) worker, findWorker returns DaemonSessionRecoveringError with wire errorInfo.code: session_recovering (daemon schema 27). Failed workers still answer Unknown active session so clients can fall through to create/reclaim. resolveActiveSessionLookupFailure and the agents view treat recovering like unknown for the saved-session reopen path, while explicit attach can surface the typed retryable error.

Reviewed by Cursor Bugbot for commit 77b747a. Bugbot is set up for automated code reviews on this repo. Configure here.

LOC

Total src: +148/−52 (net +96); tests: +260/−1 (net +259).

Note

Retry failed daemon workers on touch and add DaemonSessionRecoveringError to protocol revision 27

  • Failed workers whose recorded process identity is still current are now retried on attach, create, retry, and command forwarding via a shared retryWorkerRecovery helper that clears stop/archive markers and waits for recovery
  • Adds a typed DaemonSessionRecoveringError with an active session identifier, serialized over the wire as a structured error variant; clients deserialize it distinctly from unknown-session failures so they no longer treat a known recovering session as absent
  • findWorker resolves descriptor-known recovering root sessions to the typed error instead of unknown-session, preserving the failed-session create fallback
  • resolveActiveSessionLookupFailure in main.ts classifies structured recovering failures as thrown errors while keeping the saved-session fallback for unknown sessions; agents-view.openAgentsViewSession falls back to the saved session on recovering attach failures
  • Risk: daemon protocol schema advances to revision 27; older clients without deserializeDaemonError support see only the plain message and cannot distinguish recovering sessions. Stop-marked or identity-invalid failed workers are excluded from retry by canRetryFailedWorker — reviewers should confirm no lifecycle path clears stop markers unintentionally

Macroscope summarized 77b747a.

…r roster gaps with a structured recovering error

The recovery ladder parked a live-but-silent worker lifecycle='failed'
after its deferred probe rounds ran out, and nothing but the manual
retry_worker command ever exited that state: create reuse hard-rejected
failed workers, and only client-owned workers auto-recovered on attach.
A worker frozen longer than the probe budget (dark wake, paused VM,
heavy swap-in) was written off permanently even though its process
identity was verified current.

Failed is no longer terminal for an identity-verified live worker:
attach, create reuse, and retry_worker all share one retryWorkerRecovery
path, and the client-owned attach special case becomes the general
descriptor-matched pre-pass. Workers whose identity is gone or
unverifiable keep the old failure answer.

Session addressability gets the same single-truth treatment: findWorker
resolved selectors only through the in-memory roster, which is hydrated
lazily from live workers after adoption, so every supervisor replacement
re-opened a window where known sessions answered "Unknown active
session". On a roster miss the supervisor now consults the persisted
worker descriptors it already holds and answers with a structured,
retryable session_recovering error (new DaemonErrorInfo variant, schema
revision 27). Failed workers deliberately keep the unknown answer so
first-party clients fall back to the create path, which reclaims or
retries them; main.ts and the agents view treat the recovering error
like that fallback.
Comment thread packages/coding-agent/src/main.ts Outdated
Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7f94df0. Configure here.

Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts Outdated
…ng to explicit attach

Round-1 review fixes on the failed-worker recovery PR:

- canRetryFailedWorker now owns the stop-marker exclusion: a user-stopped
  worker (intentionalStop or persisted stopRequestedAt) is never revived
  by attach or create reuse; only the explicit retry_worker command
  clears a stop. The in-memory stop-count guard alone did not survive a
  supervisor restart, where the markers arrive from disk.
- The generalized attach pre-pass carries the owner-only payload
  (telemetry gate, launchEnv adoption, recovery context) inside the
  owned branch again; shared workers get only the descriptor lookup and
  the failed-retry gate.
- An explicit --attach-agent on a recovering session now surfaces the
  typed retryable error instead of exiting with "No active agent found":
  the get_state lookup failure classification moves into an exported
  resolveActiveSessionLookupFailure, throwing for session_recovering and
  keeping undefined (saved-session fallback) for unknown.
…r per touch; active id on the recovering wire error

Fresh-eyes review fixes on the failed-worker recovery PR:

- A parked worker keeps its roster row (parking marks rows failed), so
  findWorker matched it and forwardToWorker threw the untyped lifecycle
  error before any attach could retry recovery. The --attach-agent
  preflight's get_state died there, making the on-touch recovery
  unreachable for the primary first-party path. forwardToWorker now runs
  the same canRetryFailedWorker/retryWorkerRecovery owner as attach and
  create, completing the promised attach/prompt/create touch surface.
- One touch runs at most one recovery ladder: the post-match attach
  retry now skips the descriptor worker the pre-pass already retried,
  instead of resetting counters and running the ladder twice when
  recovery exhausted and re-parked the worker.
- The session_recovering wire error now always carries the worker's
  rootActiveSessionId; a rootSessionId selector no longer mislabels a
  stable session id as an active-session id.
Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts Outdated
…ecovering

The descriptor fallback in findWorker matched full ids only, while roster
matching also resolves unambiguous hex session-id suffixes; a
suffix-addressed root in a roster gap therefore answered Unknown instead
of the retryable session_recovering. The fallback now applies the same
matchesSessionIdSuffix rule to both descriptor ids, exact matches first;
ambiguous suffixes stay unknown.
Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts
…rker recovery

When one forwarded command started retryWorkerRecovery, the lifecycle
left 'failed' and a concurrent forwardToWorker on the same worker
skipped the retry gate straight into requireAvailableWorkerClient,
throwing 'Session worker is recovering' mid-ladder for a recovery that
was about to succeed. Join the existing worker.recovery promise instead
— the same rule create reuse already follows — preserving one ladder per
worker. The attach path deliberately keeps its fast typed failure: the
client reconnect loop owns those retries.
Shared retryableWorkerFixture/retrySupervisor builders replace repeated
scaffolding across the six recovery pins, and multi-line comment blocks
collapse to one-line invariant guards. No behavior or coverage change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant