fix(sandbox): wait through managed container restart transitions

[jyaunches]

Summary

Stopped OpenClaw and Hermes sandbox recovery now waits through exact managed-container restart transitions before declaring failure. A transition still cannot count as success: NemoClaw requires a later authenticated controller completion and the existing health, readiness, and forward checks, while persistent or diagnostic failures remain terminal.

Related Issue

Follow-up to #8726

Changes

• Retry managed recovery after status 137 only when both output streams are blank, and after Docker's canonical container-restarting result only when its 64-character lowercase container ID matches the selected registry-owned container.
• Keep one 11-attempt total recovery bound, stop after 3 exact SUPERVISOR_BUSY results, and use the existing 11-attempt read-only supervisor waiter for the two container transitions.
• Keep ordinary startup waiter entry restricted to exact SUPERVISOR_NOT_RUNNING. Status 137, Docker restart results, mismatched IDs, altered text, and diagnostic output remain terminal outside their two bounded managed-control loops.
• Cover successful transitions, persistent bounds, cumulative busy limits, mismatched IDs, altered diagnostics, startup separation, and terminal settle behavior.
• Document the recovery bounds and troubleshooting guidance for OpenClaw and Hermes.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent Codex Desktop review applied all nine repository security categories to commit 4f7614af6 and returned PASS with no findings. It verified the root-only controller boundary, exact container-ID binding, bounded retries, terminal diagnostics, and required post-transition health gates.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: Updated docs/reference/commands.mdx and docs/reference/troubleshooting.mdx.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Focused CLI and integration Vitest run passed 5 files and 173 tests; independent exact-final-commit security validation passed 13 recovery-related files and 260 tests. npm run typecheck:cli and npm run checks:repository passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable; this is a narrow sandbox lifecycle result-classification change protected by focused tests.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — npm run docs passed with 0 errors and 2 pre-existing Fern warnings.
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

E2E Evidence

• Candidate f917c46c5 reached the stopped-Hermes restart phase in run 31440989310 and exposed Docker's exact Container <id> is restarting transition. Setup and registered-resource cleanup passed.
• Trusted E2E for replacement commit 4f7614af6 is pending.

---

Signed-off-by: Julie Yaunches jyaunches@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved managed gateway recovery for empty status 137 responses and verified container-restart transitions.
  • Automatically retries recognized transient recovery states with bounded attempts, including up to 11 controller attempts and three busy responses.
  • Preserves immediate failure for invalid container IDs, unexpected diagnostics, and unrelated errors.
  • Added coverage for successful retries, retry exhaustion, and diagnostic validation.

• Documentation

  • Expanded recovery and troubleshooting guidance for transient controller states, retry limits, and container startup handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 60156ea5-faab-4870-8fc6-a436ce6d245f

📥 Commits

Reviewing files that changed from the base of the PR and between 4f7614a and 41560fc.

📒 Files selected for processing (2)

• docs/reference/commands.mdx
• docs/reference/troubleshooting.mdx

🚧 Files skipped from review as they are similar to previous changes (2)

• docs/reference/troubleshooting.mdx
• docs/reference/commands.mdx

---

📝 Walkthrough

Walkthrough

Changes

Managed supervisor recovery

Layer / File(s)	Summary
Restart failure classification src/lib/actions/sandbox/process-recovery.ts, test/process-recovery-primitives.test.ts, src/lib/actions/sandbox/start.test.ts	Supervisor actions return validated managed-control results. Exact container-restart errors require the selected container ID, status 1, empty stdout, and no extra stderr. Status-137 failures with diagnostics remain terminal.
Supervisor settling probe src/lib/actions/sandbox/process-recovery.ts, test/process-recovery-supervisor-relaunch.test.ts	The supervisor waiter retries exact empty-output status-137 results and matching container-restart transitions within the shared attempt bound.
Startup recovery retry and contract src/lib/actions/sandbox/process-recovery.ts, test/process-recovery-managed-controller.test.ts, docs/reference/commands.mdx, docs/reference/troubleshooting.mdx	Managed recovery allows up to 11 attempts, limits SUPERVISOR_BUSY to three retries, and retries validated status-137 and container-restart results. Documentation describes the retry limits and terminal cases.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ManagedRecovery
  participant SupervisorAction
  participant DockerContainer
  ManagedRecovery->>SupervisorAction: execute managed supervisor action
  SupervisorAction->>DockerContainer: probe or start controller
  DockerContainer-->>SupervisorAction: status 137 or container-restarting error
  SupervisorAction-->>ManagedRecovery: classified transient result
  ManagedRecovery->>SupervisorAction: retry within bounded attempts
  SupervisorAction-->>ManagedRecovery: completion marker or terminal failure

Loading

Possibly related PRs

• NVIDIA/NemoClaw#8683: Both PRs modify managed recovery behavior and supervisor results.
• NVIDIA/NemoClaw#8728: Both PRs modify managed supervisor recovery and waitForManagedGatewaySupervisor.

Suggested labels: integration: openclaw, integration: hermes, area: sandbox, bug-fix, v0.0.106

Suggested reviewers: cv, sandl99

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: waiting through managed container restart transitions during sandbox recovery.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-hermes-shields-restart-137

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 41560fc in the codex/fix-hermes-shi... branch remains at 96%, unchanged from commit 4298c03 in the main branch.

---

_{Updated August 11, 2026 01:07 UTC}

[coderabbitai]

🧹 Nitpick comments (2)

test/process-recovery-supervisor-relaunch.test.ts (2)

173-190: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert both bounded probe attempts.

The test verifies one sleep, but it does not prove that the waiter made the second probe before returning false. Store the request mock and assert toHaveBeenCalledTimes(2) for the configured two-attempt bound.

Proposed assertion

   it("fails after the bounded wait when empty exit 137 persists (`#8726`)", () => {
     const sleepImpl = vi.fn();
+    const requestGatewaySupervisorActionImpl = vi.fn(() => ({
+      status: 137,
+      stdout: "",
+      stderr: "",
+    }));

     expect(
       waitForManagedGatewaySupervisor("new-clone", {
         intervalSeconds: 3,
         maxAttempts: 2,
-        requestGatewaySupervisorActionImpl: vi.fn(() => ({
-          status: 137,
-          stdout: "",
-          stderr: "",
-        })),
+        requestGatewaySupervisorActionImpl,
         sleepImpl,
       }),
     ).toBe(false);
+    expect(requestGatewaySupervisorActionImpl).toHaveBeenCalledTimes(2);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/process-recovery-supervisor-relaunch.test.ts` around lines 173 - 190,
Update the test around waitForManagedGatewaySupervisor to store the
requestGatewaySupervisorActionImpl mock and assert it was called twice,
confirming both configured probe attempts occur before returning false; retain
the existing sleep assertions.

---

150-159: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a whitespace-only probe case.

The predicate accepts whitespace when both streams become empty after trim(), but this test covers only literal empty strings. Add a whitespace-only input case to protect the startup contract.

Proposed test extension

-  it("waits through an exact empty exit 137 from a settling controller probe (`#8726`)", () => {
+  it.each([
+    { stdout: "", stderr: "" },
+    { stdout: " \t", stderr: "\r\n" },
+  ])(
+    "waits through an empty or whitespace-only exit 137 from a settling controller probe (`#8726`)",
+    ({ stdout, stderr }) => {
     const sleepImpl = vi.fn();
     const requestGatewaySupervisorActionImpl = vi
       .fn()
-      .mockReturnValueOnce({ status: 137, stdout: "", stderr: "" })
+      .mockReturnValueOnce({ status: 137, stdout, stderr })
       .mockReturnValueOnce({
         status: 0,
         stdout: "GATEWAY_PID=4242",
         stderr: "",
       });

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/process-recovery-supervisor-relaunch.test.ts` around lines 150 - 159,
Add a test case alongside the exact-empty exit 137 scenario for the controller
probe, using whitespace-only stdout and stderr on the first
requestGatewaySupervisorActionImpl response. Keep the expected behavior and
subsequent successful probe response unchanged, verifying the startup flow waits
and retries when both streams become empty after trimming.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/process-recovery-supervisor-relaunch.test.ts`:
- Around line 173-190: Update the test around waitForManagedGatewaySupervisor to
store the requestGatewaySupervisorActionImpl mock and assert it was called
twice, confirming both configured probe attempts occur before returning false;
retain the existing sleep assertions.
- Around line 150-159: Add a test case alongside the exact-empty exit 137
scenario for the controller probe, using whitespace-only stdout and stderr on
the first requestGatewaySupervisorActionImpl response. Keep the expected
behavior and subsequent successful probe response unchanged, verifying the
startup flow waits and retries when both streams become empty after trimming.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 00bbe2c0-bd98-4149-bcf5-8d0a1cf917af

📥 Commits

Reviewing files that changed from the base of the PR and between 430406f and f2b5f20.

📒 Files selected for processing (6)

• src/lib/actions/sandbox/gateway-restart.test.ts
• src/lib/actions/sandbox/gateway-restart.ts
• src/lib/actions/sandbox/process-recovery.ts
• src/lib/actions/sandbox/start.test.ts
• src/lib/actions/sandbox/start.ts
• test/process-recovery-supervisor-relaunch.test.ts

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 0 warnings · 0 suggestions

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

3 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• define — container-transition result at docs/reference/commands.mdx:1422: Keep the preceding list that defines the two accepted results before using this collective term.
• justified — managed supervisor startup waiter at docs/reference/commands.mdx:1422: Keep the modifier because it distinguishes the startup probe loop from the other managed recovery loops.
• justified — managed-control loop at src/lib/actions/sandbox/process-recovery.ts:384: Keep the modifier because it distinguishes privileged managed-control retries from read-only probe loops.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: onboard-repair, onboard-resume

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

🧹 Nitpick comments (1)

test/process-recovery-managed-controller.test.ts (1)

187-221: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cover whitespace-only output, not only empty output.

These recovery cases use stdout: "" and stderr: "". They do not prove the .trim() === "" contract for non-empty whitespace. Add whitespace-only output to both a transient-success case and a persistent-failure case. Otherwise, a regression that retries only zero-length output can pass the tests.

As per path instructions, the test should prove the observable recovery behavior.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/process-recovery-managed-controller.test.ts` around lines 187 - 221,
Update the recovery test cases around the “status 137 with no output followed by
authenticated recovery” and “persistent status 137 with no output” scenarios to
use non-empty whitespace in stdout and stderr. Preserve their existing
expectedResult and expectedActions so the tests verify whitespace-only output
follows the same transient-recovery and persistent-failure behavior as empty
output.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/process-recovery-managed-controller.test.ts`:
- Around line 187-221: Update the recovery test cases around the “status 137
with no output followed by authenticated recovery” and “persistent status 137
with no output” scenarios to use non-empty whitespace in stdout and stderr.
Preserve their existing expectedResult and expectedActions so the tests verify
whitespace-only output follows the same transient-recovery and
persistent-failure behavior as empty output.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b21e9201-cf6e-4034-a993-22c8a10543d8

📥 Commits

Reviewing files that changed from the base of the PR and between f2b5f20 and f917c46.

📒 Files selected for processing (4)

• src/lib/actions/sandbox/process-recovery.ts
• src/lib/actions/sandbox/start.test.ts
• test/process-recovery-managed-controller.test.ts
• test/process-recovery-supervisor-relaunch.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/process-recovery-supervisor-relaunch.test.ts

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-8765.docs.buildwithfern.com/nemoclaw