Skip to content

fix(status): read sandbox OOM counter from the host - #7567

Merged
apurvvkumaria merged 10 commits into
mainfrom
fix/5796-host-side-oom-probe
Jul 31, 2026
Merged

fix(status): read sandbox OOM counter from the host#7567
apurvvkumaria merged 10 commits into
mainfrom
fix/5796-host-side-oom-probe

Conversation

@nvshaxie

@nvshaxie nvshaxie commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

nemoclaw status reported Phase: Ready with exit 0 and no rebuild hint after an OOM kill inside a terminal-runtime sandbox because the cgroup probe ran under the sandbox policy that denies /sys/fs/cgroup. This reads the sandbox container's counter through a bounded host-side docker exec, allowing the existing degraded-status contract to receive the signal it was designed to report.

Related Issue

Fixes #5796

Changes

  • Resolve the sandbox container through the existing OpenShell managed-by and sandbox-name labels, then apply the existing registry-backed longest-owner rule.
  • Run the fixed cgroup v2/v1 counter script through the Docker adapter with a five-second timeout.
  • Fail closed to unavailable for non-Docker drivers, registry failures, missing or ambiguous labeled containers, unresolved ownership, Docker execution failures, and malformed counter output.
  • Update unit and CLI fixtures to exercise labeled Docker discovery and host-side execution while retaining the existing text warning, rebuild guidance, JSON health result, and exit 1 assertions.

This replaces the ineffective in-sandbox probe transport. It does not add a fallback, compatibility layer, command, option, or configuration surface.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: commands, flags, JSON fields, text output, exit behavior, recovery guidance, and supported workflows are unchanged. Existing docs already describe the terminal-runtime degraded OOM contract.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop reviewed exact head b3ea62eb6; the later ee41ccaf5 head is a clean main merge that leaves that reviewed diff byte-identical. All nine security categories pass with no findings; diff fingerprint 352616bc3e775c44454432d5a7cd8b0e73111da5259e7242778dd5cf5b10ed67. The probe uses argv-style Docker execution with a fixed shell script, exact label filters, unique-container enforcement, registry-backed longest-owner validation, a five-second timeout, and fail-closed errors. It reads one counter and mutates no sandbox state.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Exact-head review of the five changed source, test, and architecture-budget files confirmed that docs/reference/commands.mdx already documents the terminal-runtime degraded OOM result, nonzero exit, structured JSON details, and rebuild guidance. Re-verified at head ee41ccaf5: the main sync is a clean merge that leaves the PR diff at the same five files, touches no .md/.mdx or docs/ path, and changes no user-visible surface. No documentation or code sample changed; no docs build was required.
  • Agent: Codex Desktop documentation writer subagent (original review at b3ea62eb6); Claude Code primary agent re-verified the unchanged scope at ee41ccaf5

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: npx vitest run --project cli src/lib/actions/sandbox/terminal-runtime-health.test.ts → 12/12; npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.ts → 26/26; npx vitest run --project e2e-support test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts -t "carries the generated planner matrix" → 1/1; npm run build:cli, npm run checks, and npm run typecheck:cli passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: not applicable; the change is scoped to one probe module, focused unit/CLI contracts, and the architecture ratchet it improves. Exact-head repository checks and normal commit/pre-push hooks passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Verification boundary

The current exact head is ee41ccaf5. It is signed, GitHub-Verified, and synced with main da1b103121039fb2e056d41b901322f05489e479. Fresh exact-head CI/E2E and automated review are running.

The physical Docker evidence remains valid but partial: in a cgroup v2 memory-limited container, an OOM-killed child exited 137 while the supervisor survived; host-side docker exec observed oom_kill change from 0 to 1; a neighboring container remained at 0; and the shipped parser classified the victim as degraded and the neighbor as ok.

Full nemoclaw status verification against a real OpenShell Docker-driver sandbox is still outstanding. The available maintainer host has OpenShell 0.0.85 but no configured gateway or registered sandbox. Approval therefore still requires exact-head OpenShell evidence showing the host counter, text and JSON degraded results, rebuild guidance, and nonzero exit, or an explicit maintainer waiver.


Signed-off-by: Shawn Xie shaxie@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved terminal runtime OOM detection by executing the probe inside the resolved sandbox container and reading cgroup metrics directly.
    • Refined runtime health classification and diagnostics for degraded/unavailable scenarios, including unsupported drivers, registry lookup issues, missing containers, ambiguous ownership, and exec failures.
  • Tests
    • Reworked and expanded OOM probe tests to cover degraded and multiple unavailability cases.
    • Updated CLI sandbox status tests to source OOM-kill data via the Docker-based probe simulation.
  • Chores
    • Adjusted the source architecture budget setting.

The cgroup OOM probe ran through the sandbox exec transport, so it
executed under the sandbox policy that denies /sys/fs/cgroup. Every read
failed and the probe returned `unavailable`, which neither status
consumer escalates on — so a real OOM kill was indistinguishable from
"the probe could not run" and `nemoclaw status` kept reporting
Phase: Ready with exit 0 and no rebuild hint.

Read the counter from the host instead. #5818 added the
terminalRuntimeHealth field, the degraded rendering, and the non-zero
exit, but left the probe on the in-sandbox transport, so the counter
those three depend on was never readable on a policy-enforced sandbox.

Resolve the sandbox's container through the existing labeled-container
discovery and longest-owner rule, then run the same probe script through
`docker exec`, which enters the container's cgroup namespace without
inheriting the sandbox policy. Ambiguous or unresolved ownership stays
`unavailable` rather than risk reporting a healthy sandbox as degraded
from another container's counter.

Fixes #5796

Signed-off-by: Shawn Xie <shaxie@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The terminal runtime cgroup OOM probe now uses host-side Docker execution, dependency injection, driver and container ownership checks, timeout-aware error handling, and expanded unit and CLI status tests.

Changes

Terminal runtime OOM probe

Layer / File(s) Summary
Docker container probe execution
src/lib/actions/sandbox/terminal-runtime-health.ts, ci/source-architecture-budget.json
The probe validates the Docker driver, resolves a unique sandbox container, executes the cgroup script through docker exec, and maps failures to unavailable results.
Probe behavior coverage
src/lib/actions/sandbox/terminal-runtime-health.test.ts
Tests cover command construction, degraded OOM classifications, execution failures, unsupported drivers, missing containers, ambiguous ownership, unresolved owners, and registry failures.
CLI status integration
test/cli/sandbox-status-json.test.ts, test/cli/sandbox-status-text.test.ts
CLI fixtures now emit OOM counters and cgroup source data through mocked Docker execution while OpenShell execution returns a success response.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Probe as probeTerminalRuntimeCgroupOom
  participant Registry as readSandboxDriver
  participant Resolver as resolveSandboxContainerOwner
  participant Docker as dockerSpawnSync
  participant Status as CLI status
  Probe->>Registry: Read sandbox driver
  Registry-->>Probe: Return docker
  Probe->>Resolver: Resolve sandbox container
  Resolver-->>Probe: Return container name
  Probe->>Docker: Execute cgroup OOM probe
  Docker-->>Probe: Return OOM counters and source
  Probe-->>Status: Return terminal runtime health
Loading

Suggested labels: platform: container

Suggested reviewers: cv, aasthajh, caroline-xuan

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes address #5796 by detecting host-side OOM conditions, surfacing degraded status and rebuild guidance, and preserving nonzero exit behavior.
Out of Scope Changes check ✅ Passed The diff appears focused on the OOM-status fix and related test/CI adjustments, with no clear unrelated feature work.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately summarizes the main change: reading sandbox OOM counters from the host for status health checks.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/5796-host-side-oom-probe

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit ee41cca in the fix/5796-host-side-o... branch remains at 96%, unchanged from commit 376beb5 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit ee41cca in the fix/5796-host-side-o... branch remains at 81%, unchanged from commit 376beb5 in the main branch.

Show a code coverage summary of the most impacted files.
File main 376beb5 fix/5796-host-side-o... ee41cca +/-
src/lib/actions...time-command.ts 100% 82% -18%
src/lib/actions...-add-restart.ts 19% 10% -9%
src/lib/domain/.../connect-env.ts 97% 89% -8%
src/lib/actions...lution-probe.ts 95% 88% -7%
src/lib/actions...x/mcp-bridge.ts 41% 35% -6%
src/lib/onboard...shboard-port.ts 96% 90% -6%
src/lib/actions...ntime-health.ts 91% 85% -6%
src/lib/actions...e-validation.ts 84% 81% -3%
src/lib/actions...dbox/destroy.ts 95% 93% -2%
src/lib/onboard...eway-service.ts 82% 81% -1%

Updated July 30, 2026 05:17 UTC

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/actions/sandbox/terminal-runtime-health.test.ts`:
- Line 54: Update the test title in the OOM counter test to append the issue
reference as the final “(`#5796`)” suffix, and remove the standalone body comment
reference if present.

In `@src/lib/actions/sandbox/terminal-runtime-health.ts`:
- Line 103: Update listSandboxNames in the registry-backed driver so registry
failures are propagated as an explicit unavailable state rather than thrown or
replaced with an empty list, preserving exact-name validation safety. Ensure
probeTerminalRuntimeCgroupOom handles that failure consistently with
readSandboxDriver, and add a test using a throwing registry that verifies status
evaluation returns unavailable.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 88610e4e-5da2-489e-8288-8d19aa2d1cf7

📥 Commits

Reviewing files that changed from the base of the PR and between 3e930f2 and 412f010.

📒 Files selected for processing (2)
  • src/lib/actions/sandbox/terminal-runtime-health.test.ts
  • src/lib/actions/sandbox/terminal-runtime-health.ts

Comment thread src/lib/actions/sandbox/terminal-runtime-health.test.ts Outdated
Comment thread src/lib/actions/sandbox/terminal-runtime-health.ts Outdated
@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume

1 optional E2E recommendation
  • ubuntu-repo-docker-post-reboot-recovery

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@cv cv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The host-side transport is directionally correct and the focused unit suite passes 11/11, but this exact head is not release-ready:

  1. The checked-in CLI OOM contracts still stub the removed OpenShell exec path. On this exact head, after npm run build:cli, npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.ts fails both #5796 cases: JSON gets { kind: "unavailable", detail: "no labeled container found for sandbox" } instead of degraded, and text exits 0 instead of 1. Update the fixtures to prove labeled docker ps discovery and host-side docker exec output, while retaining the user-visible warning/rebuild/exit assertions.
  2. defaultDeps.listSandboxNames calls registry.listSandboxes() without a guard. A registry read error therefore escapes from probeTerminalRuntimeCgroupOom and can crash status, contradicting the fail-closed unavailable contract. Propagate/handle that failure explicitly and add a throwing-registry regression.
  3. This is a sandbox-boundary change for a QA-escaped defect, and the PR explicitly says the real OpenShell status path was not verified. Please attach exact-head Docker-driver/OpenShell OOM evidence showing the host counter, text/JSON degraded result, rebuild hint, and nonzero exit, or record a maintainer waiver. Also complete the exact-head sensitive-path and documentation-writer receipts before final gates.

Small convention cleanup: append (#5796) to the changed behavior-oriented test title instead of keeping the issue reference only in a body comment. After the fixes, refresh once onto current main and rerun exact-head CI/E2E.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
…er-followup

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator

Addressed the concrete code and test findings in 8658fbe2b, then synced current main in signed, Verified exact head 9d9fd0184.

  • routes the host cgroup probe through the Docker adapter
  • converts registry read failures to explicit unavailable without unsafe exact-name fallback
  • updates the text/JSON CLI fixtures to exercise labeled docker ps discovery plus host-side docker exec, retaining degraded output, rebuild guidance, and exit 1 assertions
  • moves (#5796) to the behavior-oriented test title

Local validation: terminal-runtime unit tests 12/12; focused JSON and text OOM status tests 1/1 each; Docker abstraction guard 1/1; CLI build/typecheck; all repository hooks; pre-push checks. Documentation-writer review found no docs change required. The two CodeRabbit threads are resolved.

Still outstanding before merge readiness: fresh exact-head CI/sensitive-path receipts and the requested real Docker-driver/OpenShell OOM evidence (or an explicit maintainer waiver).

@nvshaxie

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. Status at head 9d9fd018, verified on that exact head.

Note on provenance: commits 8658fbe2b and 9d9fd0184 were pushed to this branch by the maintainer-followup automation, not by me. I re-verified them rather than assuming they were correct.

1. CLI OOM contracts still stubbing the removed exec path — fixed. Your exact repro now passes on this head:

$ npm run build:cli
$ npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.ts
Test Files  2 passed (2)
     Tests  26 passed (26)

The fixtures now drive labeled docker ps discovery and host-side docker exec output, and the user-visible warning/rebuild/exit assertions are retained.

2. listSandboxNames unguarded registry read — fixed. readSandboxNames() now mirrors readSandboxDriver() and returns undefined on throw; the probe maps that to { kind: "unavailable", detail: "sandbox ownership registry unavailable" } rather than letting it escape into status. Throwing-registry regression added — returns unavailable when sandbox ownership registry lookup fails. Focused suite 12/12.

4. Test title convention — fixed. Now reads cgroup OOM counters from the host rather than inside the sandbox (#5796), with the body comment reference removed.

3. Live OOM evidence — partial, and I want to be explicit about the boundary.

This host has Docker but no OpenShell installed and no labeled NemoClaw sandbox containers, so I cannot produce the full OpenShell status-path evidence you asked for. I am not claiming that verification.

What I did verify is the physical assumption the fix rests on — that a host-side docker exec reads the container's own counter in the supervisor-survives case. Using the probe script extracted verbatim from CGROUP_OOM_PROBE_SCRIPT:

# 64m-limited container, PID 1 = sleep, child hog OOM-killed
BEFORE:  oom_kill=0  source=/sys/fs/cgroup/memory.events.local
child exit: 137
supervisor: Running=true  OOMKilled=true      <-- the #5796 shape
AFTER:   oom_kill=1  source=/sys/fs/cgroup/memory.events.local  (probe exit 0)

Counter isolation, which is what makes the ambiguous-ownership fail-closed rule matter:

neighbor container:  oom_kill=0
victim container:    oom_kill=1

And the real output through the shipped parser:

victim   -> {"kind":"degraded","oomKillCount":1,"source":"/sys/fs/cgroup/memory.events.local"}
neighbor -> {"kind":"ok","oomKillCount":0,"source":"/sys/fs/cgroup/memory.events.local"}

So the kernel counter, the probe script, and the parser agree end-to-end on the degraded verdict, and the counter is per-container rather than host-wide. What remains unproven on my hardware is the OpenShell-driver leg — sandbox registry lookup and label discovery against a real provisioned sandbox. That needs a maintainer waiver or a run on a host with OpenShell; I'd rather flag it than imply coverage I don't have.

CI. The only genuine failure on this head was reviewed-npm-audit, which failed with:

npm audit failed without a complete vulnerability report:
npm audit report metadata must be an object; {"summary":"","detail":""}

That's an empty/malformed audit report rather than an advisory, and this PR changes no dependency files (only the four source/test files above). checks failed solely because it requires reviewed-npm-audit, and E2E / PR Gate timed out waiting on checksstatic-checks, build-typecheck, cli-tests, plugin-tests, and installer-integration all passed. I've re-run the failed jobs.

CI / Documentation Writer Review is passing on this head. Happy to take direction on whether the sensitive-path review still needs a separate explicit receipt given this is a probe-transport change that reads one counter and mutates nothing.

Picks up #7575 (ci(security): retry incomplete npm audits), which addresses
the transient 'npm audit report metadata must be an object' failure that
blocked reviewed-npm-audit on this branch. Refresh requested in review of #7567.
@nvshaxie

Copy link
Copy Markdown
Contributor Author

Addendum on the CI point: re-running alone was the wrong fix. This branch was 27 commits behind main and did not contain #7575 (ci(security): retry incomplete npm audits), which is exactly the retry for the npm audit report metadata must be an object failure mode that blocked reviewed-npm-audit here.

Refreshed onto current main in 7fba55d6 — which also covers your "refresh once onto current main and rerun exact-head CI/E2E". Merge was clean, and both suites still pass on the merged head after npm run build:cli: focused 12/12, CLI OOM contracts 26/26.

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
@nvshaxie

Copy link
Copy Markdown
Contributor Author

Receipt refresh for the new exact head. 5e2b25de (merge(main): refresh PR gate controller) landed on this branch after my last status, which was taken at 7fba55d6, so I re-ran the two suites from your review on that exact head after npm run build:cli:

$ npx vitest run --project cli src/lib/actions/sandbox/terminal-runtime-health.test.ts
Test Files  1 passed (1)   Tests  12 passed (12)

$ npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.ts
Test Files  2 passed (2)   Tests  26 passed (26)

Exact-head CI is fully green on 5e2b25de (including reviewed-npm-audit, checks, and E2E / PR Gate Coordination), both CodeRabbit threads are resolved, and the documentation-writer receipt already carries this head. Item 3 — live OpenShell-driver OOM evidence or a waiver — remains the only thing I have open from your list; no change from my previous comment.

@cjagwani cjagwani added v0.0.98 and removed v0.0.97 labels Jul 28, 2026
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@wscurran wscurran added area: cli Command line interface, flags, terminal UX, or output area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression integration: dcode LangChain Deep Code integration behavior labels Jul 28, 2026
@nvshaxie

Copy link
Copy Markdown
Contributor Author

Receipt refresh for the new exact head b3ea62eb6.

Provenance first: four commits landed on this branch after my last receipt (taken at 5e2b25de), none of them mine — 8bfda8037 (cjagwani), then aa974e6f0, 9b8d0bab4, and b3ea62eb6 (apurvvkumaria). One of them changed a checked-in ratchet file, so I re-verified this head rather than assuming the automation was correct.

Exact-head verification — after npm run build:cli on b3ea62eb6:

$ npx vitest run --project cli src/lib/actions/sandbox/terminal-runtime-health.test.ts
Test Files  1 passed (1)   Tests  12 passed (12)

$ npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.ts
Test Files  2 passed (2)   Tests  26 passed (26)

$ npx vitest run --project integration test/source-architecture.test.ts
Test Files  1 passed (1)   Tests  7 passed (7)

Required CI is green on this head. The one CANCELLED E2E / PR Gate entry is a superseded duplicate; the sibling run for the same gate is SUCCESS.

On the CodeRabbit "Out of Scope Changes" warning (ci/source-architecture-budget.json): the change is required by this fix, not an unrelated tweak. The single edited line lowers src/lib/adapters/openshell/resolve.ts fan-in from 28 to 27, because this PR removed the probe's import of that module when it moved the transport off the in-sandbox OpenShell exec path. test/source-architecture.test.ts is a strict-equality ratchet, so leaving the old value in place fails rather than passing with slack. Reverting just that line on this head reproduces:

AssertionError: expected [ { kind: 'metric-ratchet', …(4) } ] to deeply equal []
+   {
+     "actual": 27,
+     "file": "src/lib/adapters/openshell/resolve.ts",
+     "kind": "metric-ratchet",
+     "limit": 28,
+     "metric": "fan-in",
+   },

The budget line tightens a ratchet the fix earned; dropping it would red the codebase-growth-guardrails gate.

Still open — item 3 of @cv's review only. Points 1 and 2 were addressed and re-verified above. Point 3 asks for exact-head Docker-driver/OpenShell OOM evidence through real nemoclaw status, or a maintainer waiver.

I can't close that one from my side. The physical Docker half is already recorded in the PR description and still holds: in a cgroup v2 memory-limited container an OOM-killed child exited 137 while the supervisor survived, host-side docker exec observed oom_kill go 0 → 1, a neighbouring container stayed at 0, and the shipped parser classified the victim degraded and the neighbour ok. What I don't have is an OpenShell host — the machine available to me has no configured gateway or registered sandbox, so the last mile (nemoclaw status text/JSON degraded output, rebuild hint, nonzero exit against a real Docker-driver sandbox) can't be produced here.

@cv — could you either point me at a host where I can run that, or record the waiver? Everything else on this PR is done and verified at b3ea62eb6.

@cjagwani

Copy link
Copy Markdown
Collaborator

Babysitting status for exact head b3ea62e (plain comment; no Changes Requested review from me): the earlier concrete findings have follow-up receipts and all 54 current check contexts are green. The remaining gate is freshness: this head is based on c4c020c while current main is da1b103, and maintainer edits are disabled. Please refresh the branch; I will re-run the exact-head gate after the new head is quiet for one hour.

Sync fix/5796-host-side-oom-probe onto main da1b103 so the exact-head
gate runs against current main. Maintainer edits are disabled on this
PR, so the branch refresh is contributor-side.

Signed-off-by: Shawn Xie <shaxie@nvidia.com>
@nvshaxie

Copy link
Copy Markdown
Contributor Author

Branch refreshed as requested. New exact head is ee41ccaf5, a signed and GitHub-Verified merge of main da1b10312 into b3ea62eb6. The merge was clean — no conflicts, and it changes no file this PR owns.

Exact-head verification — after npm install and npm run build:cli on ee41ccaf5:

$ npx vitest run --project cli src/lib/actions/sandbox/terminal-runtime-health.test.ts
Test Files  1 passed (1)   Tests  12 passed (12)

$ npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.ts test/source-architecture.test.ts
Test Files  3 passed (3)   Tests  33 passed (33)

$ npm run typecheck:cli
(clean)

The architecture ratchet still holds on current main: src/lib/adapters/openshell/resolve.ts fan-in stays at the lowered 27 that this PR's transport change earns, and test/source-architecture.test.ts passes 7/7 under strict equality.

One note for anyone refreshing this branch locally: main now carries tools/advisors/http-dispatcher.mts, which imports undici. A checkout with stale node_modules fails the tsc-cli pre-push hook with TS2307: Cannot find module 'undici' until npm install runs. That is a local-environment step only — no source or lockfile change was needed, and I committed none.

The verification boundary in the PR description is unchanged: physical Docker evidence for the host-side counter remains valid but partial, and full nemoclaw status verification against a real OpenShell Docker-driver sandbox is still outstanding, since the host available to me has no configured gateway or registered sandbox. That open item still needs either exact-head OpenShell evidence or an explicit maintainer waiver.

@cjagwani

Copy link
Copy Markdown
Collaborator

Thanks for refreshing the branch. I learned after my request that conflict-free base refreshes are explicitly waived, so that request should not have been made solely for base currency. I will not ask for another such refresh; the new exact head is now protected by the one-hour quiet window while its CI/E2E evidence settles. This is a plain coordination comment, not Changes Requested.

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE for exact head ee41ccaf552a24891ec7c7c617eef19bfd85a178.

The earlier concrete findings are resolved: the CLI fixtures exercise labeled Docker discovery plus host-side docker exec, registry lookup failure maps to unavailable, and the test title follows the issue-suffix convention. I independently reproduced the focused probe suite (12/12), the text/JSON status suites (26/26), CLI build, and CLI typecheck. Required CI reports all 53 current checks green; CodeRabbit has no unresolved major/critical findings; all 10 commits are Verified; GitHub reports MERGEABLE.

Security review: PASS across the nine categories. The probe is read-only, uses an argv-style Docker invocation and a fixed shell program, requires a unique labeled container plus registry-backed longest-owner resolution, bounds execution to five seconds, parses only a numeric counter, exposes no secrets, and fails closed on unavailable or ambiguous state.

Scoped maintainer waiver: full live nemoclaw status evidence on a configured OpenShell Docker-driver host is waived for this patch. The recorded physical Docker experiment validates the supervisor-survives cgroup shape and per-container counter isolation; the focused integration contracts validate text/JSON degraded output, rebuild guidance, and nonzero exit; exact-head CI/E2E is green. This waiver is limited to this read-only transport change.

The deterministic gate's only non-pass is base age. Per the explicit conflict-free-refresh waiver, base currency alone is not a blocker while GitHub reports MERGEABLE; do not merge main again solely for freshness.

@cjagwani

Copy link
Copy Markdown
Collaborator

@cv The blocking review is attached to old head 412f010af; exact head ee41ccaf5 resolves the CLI OOM contract and host-side transport findings, and the focused verification is green. Could you re-review or dismiss the stale Changes Requested state? I am continuing to babysit this PR through terminal state.

@apurvvkumaria
apurvvkumaria merged commit d0b7cd6 into main Jul 31, 2026
96 of 98 checks passed
@apurvvkumaria
apurvvkumaria deleted the fix/5796-host-side-oom-probe branch July 31, 2026 17:43
@senthilr-nv senthilr-nv mentioned this pull request Aug 1, 2026
23 tasks
senthilr-nv added a commit that referenced this pull request Aug 1, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical dated changelog entry for `v0.0.100` so the
maintainer release plan can verify the pre-tag documentation
prerequisite. The entry summarizes the user-facing changes merged since
`v0.0.99` and links to the relevant guides.

## Changes

- Add `docs/changelog/2026-07-31.mdx` with the exact `## v0.0.100`
heading.
- Cover restored OpenClaw pairing, transactional replacement, Deep
Agents Code, onboarding recovery, lifecycle cleanup, Hermes builds, host
provenance, documentation, and trusted E2E evidence.
- Distinguish active Docker and Kubernetes runtime-bundle enforcement
from the still-inactive managed shared-state transaction foundation.

## Source Coverage

The release entry maps the doc-impacting merged PRs in the
`v0.0.99..main` release range to `docs/changelog/2026-07-31.mdx`: #8021,
#8024, #7973, #8028, #7947, #7788, #7884, #8023, #7969, #8020, #7989,
#8000, #7907, #7942, #7567, #8013, #7955, #8017, #8014, #8015, #7629,
#7644, #7821, #7971, and #7991.

PR #7974 was reviewed after the final rebase and excluded because it
changes internal maintainer-skill attribution policy and tests only; it
does not change a user-facing product or documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification: the
changelog contract test validates the dated entry, version heading, SPDX
form, and route constraints.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/changelog/2026-07-31.mdx`; exact-head review passed
for `6093f44f`; writing rules and documentation style reviewed; `npx
vitest run test/changelog-docs.test.ts` passed 6/6; `npm run docs`
passed with zero Fern errors and two generic Fern upgrade notices.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 6093f44 -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable; no DGX Station host script changed.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npx
vitest run test/changelog-docs.test.ts` passed 6/6 at `6093f44f`.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to a dated
prose-only release entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
validation passed with zero errors; Fern emitted two generic upgrade
notices.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the changelog entry has the required parser-safe MDX SPDX header;
dated changelog entries intentionally do not use page frontmatter.

---

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Added release notes for v0.0.100.
* Documented improvements to restore pairing, sandbox replacement,
onboarding recovery, lifecycle cleanup, runtime handling, build support,
host readiness, and end-to-end validation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli Command line interface, flags, terminal UX, or output area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression integration: dcode LangChain Deep Code integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Linux][CLI&UX] nemoclaw status does not show Error phase or rebuild hint after OOM kills dcode process inside sandbox

6 participants