fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error - #2047
Open
snimu wants to merge 7 commits into
Open
fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error#2047snimu wants to merge 7 commits into
snimu wants to merge 7 commits into
Conversation
…r roster gaps with a structured recovering error The recovery ladder parked a live-but-silent worker lifecycle='failed' after its deferred probe rounds ran out, and nothing but the manual retry_worker command ever exited that state: create reuse hard-rejected failed workers, and only client-owned workers auto-recovered on attach. A worker frozen longer than the probe budget (dark wake, paused VM, heavy swap-in) was written off permanently even though its process identity was verified current. Failed is no longer terminal for an identity-verified live worker: attach, create reuse, and retry_worker all share one retryWorkerRecovery path, and the client-owned attach special case becomes the general descriptor-matched pre-pass. Workers whose identity is gone or unverifiable keep the old failure answer. Session addressability gets the same single-truth treatment: findWorker resolved selectors only through the in-memory roster, which is hydrated lazily from live workers after adoption, so every supervisor replacement re-opened a window where known sessions answered "Unknown active session". On a roster miss the supervisor now consults the persisted worker descriptors it already holds and answers with a structured, retryable session_recovering error (new DaemonErrorInfo variant, schema revision 27). Failed workers deliberately keep the unknown answer so first-party clients fall back to the create path, which reclaims or retries them; main.ts and the agents view treat the recovering error like that fallback.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7f94df0. Configure here.
…ng to explicit attach Round-1 review fixes on the failed-worker recovery PR: - canRetryFailedWorker now owns the stop-marker exclusion: a user-stopped worker (intentionalStop or persisted stopRequestedAt) is never revived by attach or create reuse; only the explicit retry_worker command clears a stop. The in-memory stop-count guard alone did not survive a supervisor restart, where the markers arrive from disk. - The generalized attach pre-pass carries the owner-only payload (telemetry gate, launchEnv adoption, recovery context) inside the owned branch again; shared workers get only the descriptor lookup and the failed-retry gate. - An explicit --attach-agent on a recovering session now surfaces the typed retryable error instead of exiting with "No active agent found": the get_state lookup failure classification moves into an exported resolveActiveSessionLookupFailure, throwing for session_recovering and keeping undefined (saved-session fallback) for unknown.
…r per touch; active id on the recovering wire error Fresh-eyes review fixes on the failed-worker recovery PR: - A parked worker keeps its roster row (parking marks rows failed), so findWorker matched it and forwardToWorker threw the untyped lifecycle error before any attach could retry recovery. The --attach-agent preflight's get_state died there, making the on-touch recovery unreachable for the primary first-party path. forwardToWorker now runs the same canRetryFailedWorker/retryWorkerRecovery owner as attach and create, completing the promised attach/prompt/create touch surface. - One touch runs at most one recovery ladder: the post-match attach retry now skips the descriptor worker the pre-pass already retried, instead of resetting counters and running the ladder twice when recovery exhausted and re-parked the worker. - The session_recovering wire error now always carries the worker's rootActiveSessionId; a rootSessionId selector no longer mislabels a stable session id as an active-session id.
…ecovering The descriptor fallback in findWorker matched full ids only, while roster matching also resolves unambiguous hex session-id suffixes; a suffix-addressed root in a roster gap therefore answered Unknown instead of the retryable session_recovering. The fallback now applies the same matchesSessionIdSuffix rule to both descriptor ids, exact matches first; ambiguous suffixes stay unknown.
…rker recovery When one forwarded command started retryWorkerRecovery, the lifecycle left 'failed' and a concurrent forwardToWorker on the same worker skipped the retry gate straight into requireAvailableWorkerClient, throwing 'Session worker is recovering' mid-ladder for a recovery that was about to succeed. Join the existing worker.recovery promise instead — the same rule create reuse already follows — preserving one ladder per worker. The attach path deliberately keeps its fast typed failure: the client reconnect loop owns those retries.
Shared retryableWorkerFixture/retrySupervisor builders replace repeated scaffolding across the six recovery pins, and multi-line comment blocks collapse to one-line invariant guards. No behavior or coverage change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Part of the worker-state single-truth program (Linear RES-1270); squashes the confirmed residuals of discussions #1742 and #1650 and the mechanism behind #1870/#1641 (and the #1904d reconnect gap).
Purpose
Two owners for two worker-state facts:
MAX_DEFERRED_RECOVERY_ROUNDSthe recovery ladder parks a live-but-silent workerlifecycle='failed'terminally: no timer re-probes it,reuseWorkerForCreatehard-rejects it, and only client-owned workers auto-recover on attach. A worker frozen longer than the ~2-3 min probe budget (dark wake, paused VM, swap-in) stays written off even thoughisProcessAlive+processStartIdprove it is ours and alive ([Bug] Sleeping the machine permanently orphans daemon sessions: a live worker is marked failed and never re-checked #1742 residual). The create path threw exactly in the gap where neitherretry_workernor a launchEnv-bearing fresh create applied ([Bug] Prime Agent 0.8.0: saved session becomes permanently unopenable after failed worker recovery #1650 residual).findWorkerresolves selectors only through the in-memory roster, hydrated lazily from live workers after adoption; the persisted worker descriptors that carryrootActiveSessionId/rootSessionIdwere consulted only in the client-owned attach special case. Every supervisor replacement re-opens a window where attach/prompt/kill on a provably-known session answerUnknown active session, and clients cannot distinguish gone from recovering (v0.8.1: supervisor restart loses live worker active-session mapping during RLM fan-out #1870, [Bug] v0.8.0 supervisor restarts make every TUI fail to reattach with “Unknown active session” #1641).Change
canRetryFailedWorker(failed + no stop in flight + process identity verified current) andretryWorkerRecovery(the state resetretry_workeralready performed) become the single retry semantics.retry_worker, attach, and create reuse all share them; the parked state is now exit-able on any touch. Identity gone/replaced/unverifiable keeps the old hard answers.findWorkeron a roster miss consults descriptors: a root session on a recovery-candidate worker answers with a structured, retryablesession_recoveringerror (DaemonSessionRecoveringError, newDaemonErrorInfovariant,DAEMON_SCHEMA_REVISION27). Failed workers deliberately keepUnknown active sessionso first-party clients take the saved-session/create fallback, which reclaims or retries them (that fallback is what self-heals identity-gone workers vialaunchEnv).main.ts(get_state lookup) and the agents view treatsession_recoveringlike the unknown fallback, converging on the create path.Wire classification: backward-compatible. The new variant rides the existing optional
errorInfo; old clients see a readable message, old daemons never send it, nothing is capability-gated because it degrades to a plain error.Net src LOC: +102/-38, of which the supervisor is +84/-38 (about half is the owned-attach special case becoming the general pre-pass, plus the two shared helpers replacing three inline copies of the reset).
Tests
session_recovering(typed, withactiveSessionId); a failed worker and a truly unknown selector keepUnknown active session(fail-unfixed verified: removing the fallback yields Unknown).daemon-errors.test.ts: serialize/deserialize round-trip for new clients, readable message for old clients, and plain old-daemon failures do not misclassify.Ran locally: daemon-supervisor-{monitor,process,eviction,admission,input-pause}, daemon-agent-roster, daemon-supervisor-lazy-subagents, daemon-errors, regressions 4656 + 4603, agent-connection-daemon, daemon-ps, agents-view suite, main-interactive-routing — 540 tests pass;
npm run checkgreen.Note
Medium Risk
Touches core daemon supervisor routing, attach/create, and worker lifecycle; behavior changes for failed-but-live workers and session lookup errors, though guarded by process identity and extensive new tests.
Overview
Fixes daemon sessions getting stuck when a worker is parked
failedbut its process is still alive, and when the supervisor knows a session id but cannot route to it yet.Failed worker recovery: Shared
canRetryFailedWorker/retryWorkerRecoveryreplace duplicated reset logic.retry_worker, session create reuse, attach, and any command forwarded to the worker now automatically retry recovery when failure is not terminal (current process identity, no stop in flight). User-stopped workers stay stopped until an explicitretry_worker.Addressability: When roster lookup misses but a persisted descriptor matches a recovering (non-failed) worker,
findWorkerreturnsDaemonSessionRecoveringErrorwith wireerrorInfo.code: session_recovering(daemon schema 27). Failed workers still answerUnknown active sessionso clients can fall through to create/reclaim.resolveActiveSessionLookupFailureand the agents view treat recovering like unknown for the saved-session reopen path, while explicit attach can surface the typed retryable error.Reviewed by Cursor Bugbot for commit 77b747a. Bugbot is set up for automated code reviews on this repo. Configure here.
LOC
Total src: +148/−52 (net +96); tests: +260/−1 (net +259).
Note
Retry failed daemon workers on touch and add
DaemonSessionRecoveringErrorto protocol revision 27retryWorkerRecoveryhelper that clears stop/archive markers and waits for recoveryDaemonSessionRecoveringErrorwith an active session identifier, serialized over the wire as a structured error variant; clients deserialize it distinctly from unknown-session failures so they no longer treat a known recovering session as absentfindWorkerresolves descriptor-known recovering root sessions to the typed error instead of unknown-session, preserving the failed-session create fallbackresolveActiveSessionLookupFailurein main.ts classifies structured recovering failures as thrown errors while keeping the saved-session fallback for unknown sessions;agents-view.openAgentsViewSessionfalls back to the saved session on recovering attach failuresdeserializeDaemonErrorsupport see only the plain message and cannot distinguish recovering sessions. Stop-marked or identity-invalid failed workers are excluded from retry bycanRetryFailedWorker— reviewers should confirm no lifecycle path clears stop markers unintentionallyMacroscope summarized 77b747a.