fix(status): read sandbox OOM counter from the host - #7567
Conversation
The cgroup OOM probe ran through the sandbox exec transport, so it executed under the sandbox policy that denies /sys/fs/cgroup. Every read failed and the probe returned `unavailable`, which neither status consumer escalates on — so a real OOM kill was indistinguishable from "the probe could not run" and `nemoclaw status` kept reporting Phase: Ready with exit 0 and no rebuild hint. Read the counter from the host instead. #5818 added the terminalRuntimeHealth field, the degraded rendering, and the non-zero exit, but left the probe on the in-sandbox transport, so the counter those three depend on was never readable on a policy-enforced sandbox. Resolve the sandbox's container through the existing labeled-container discovery and longest-owner rule, then run the same probe script through `docker exec`, which enters the container's cgroup namespace without inheriting the sandbox policy. Ambiguous or unresolved ownership stays `unavailable` rather than risk reporting a healthy sandbox as degraded from another container's counter. Fixes #5796 Signed-off-by: Shawn Xie <shaxie@nvidia.com>
📝 WalkthroughWalkthroughThe terminal runtime cgroup OOM probe now uses host-side Docker execution, dependency injection, driver and container ownership checks, timeout-aware error handling, and expanded unit and CLI status tests. ChangesTerminal runtime OOM probe
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Probe as probeTerminalRuntimeCgroupOom
participant Registry as readSandboxDriver
participant Resolver as resolveSandboxContainerOwner
participant Docker as dockerSpawnSync
participant Status as CLI status
Probe->>Registry: Read sandbox driver
Registry-->>Probe: Return docker
Probe->>Resolver: Resolve sandbox container
Resolver-->>Probe: Return container name
Probe->>Docker: Execute cgroup OOM probe
Docker-->>Probe: Return OOM counters and source
Probe-->>Status: Return terminal runtime health
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit ee41cca in the TypeScript / code-coverage/cliThe overall coverage in commit ee41cca in the Show a code coverage summary of the most impacted files.
Updated |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/actions/sandbox/terminal-runtime-health.test.ts`:
- Line 54: Update the test title in the OOM counter test to append the issue
reference as the final “(`#5796`)” suffix, and remove the standalone body comment
reference if present.
In `@src/lib/actions/sandbox/terminal-runtime-health.ts`:
- Line 103: Update listSandboxNames in the registry-backed driver so registry
failures are propagated as an explicit unavailable state rather than thrown or
replaced with an empty list, preserving exact-name validation safety. Ensure
probeTerminalRuntimeCgroupOom handles that failure consistently with
readSandboxDriver, and add a test using a throwing registry that verifies status
evaluation returns unavailable.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 88610e4e-5da2-489e-8288-8d19aa2d1cf7
📒 Files selected for processing (2)
src/lib/actions/sandbox/terminal-runtime-health.test.tssrc/lib/actions/sandbox/terminal-runtime-health.ts
PR Review Advisor — No blocking findings reportedAdvisor assessment: No blocking advisor findings reported Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 optional E2E recommendation
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
cv
left a comment
There was a problem hiding this comment.
The host-side transport is directionally correct and the focused unit suite passes 11/11, but this exact head is not release-ready:
- The checked-in CLI OOM contracts still stub the removed OpenShell exec path. On this exact head, after
npm run build:cli,npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.tsfails both #5796 cases: JSON gets{ kind: "unavailable", detail: "no labeled container found for sandbox" }instead ofdegraded, and text exits 0 instead of 1. Update the fixtures to prove labeleddocker psdiscovery and host-sidedocker execoutput, while retaining the user-visible warning/rebuild/exit assertions. defaultDeps.listSandboxNamescallsregistry.listSandboxes()without a guard. A registry read error therefore escapes fromprobeTerminalRuntimeCgroupOomand can crashstatus, contradicting the fail-closedunavailablecontract. Propagate/handle that failure explicitly and add a throwing-registry regression.- This is a sandbox-boundary change for a QA-escaped defect, and the PR explicitly says the real OpenShell status path was not verified. Please attach exact-head Docker-driver/OpenShell OOM evidence showing the host counter, text/JSON degraded result, rebuild hint, and nonzero exit, or record a maintainer waiver. Also complete the exact-head sensitive-path and documentation-writer receipts before final gates.
Small convention cleanup: append (#5796) to the changed behavior-oriented test title instead of keeping the issue reference only in a body comment. After the fixes, refresh once onto current main and rerun exact-head CI/E2E.
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
…er-followup Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
|
Addressed the concrete code and test findings in
Local validation: terminal-runtime unit tests 12/12; focused JSON and text OOM status tests 1/1 each; Docker abstraction guard 1/1; CLI build/typecheck; all repository hooks; pre-push checks. Documentation-writer review found no docs change required. The two CodeRabbit threads are resolved. Still outstanding before merge readiness: fresh exact-head CI/sensitive-path receipts and the requested real Docker-driver/OpenShell OOM evidence (or an explicit maintainer waiver). |
|
Thanks for the detailed review. Status at head Note on provenance: commits 1. CLI OOM contracts still stubbing the removed exec path — fixed. Your exact repro now passes on this head: The fixtures now drive labeled 2. 4. Test title convention — fixed. Now 3. Live OOM evidence — partial, and I want to be explicit about the boundary. This host has Docker but no OpenShell installed and no labeled NemoClaw sandbox containers, so I cannot produce the full OpenShell status-path evidence you asked for. I am not claiming that verification. What I did verify is the physical assumption the fix rests on — that a host-side Counter isolation, which is what makes the ambiguous-ownership fail-closed rule matter: And the real output through the shipped parser: So the kernel counter, the probe script, and the parser agree end-to-end on the degraded verdict, and the counter is per-container rather than host-wide. What remains unproven on my hardware is the OpenShell-driver leg — sandbox registry lookup and label discovery against a real provisioned sandbox. That needs a maintainer waiver or a run on a host with OpenShell; I'd rather flag it than imply coverage I don't have. CI. The only genuine failure on this head was That's an empty/malformed audit report rather than an advisory, and this PR changes no dependency files (only the four source/test files above).
|
|
Addendum on the CI point: re-running alone was the wrong fix. This branch was 27 commits behind main and did not contain #7575 ( Refreshed onto current main in |
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
|
Receipt refresh for the new exact head. Exact-head CI is fully green on |
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
|
Receipt refresh for the new exact head Provenance first: four commits landed on this branch after my last receipt (taken at Exact-head verification — after Required CI is green on this head. The one On the CodeRabbit "Out of Scope Changes" warning ( The budget line tightens a ratchet the fix earned; dropping it would red the Still open — item 3 of @cv's review only. Points 1 and 2 were addressed and re-verified above. Point 3 asks for exact-head Docker-driver/OpenShell OOM evidence through real I can't close that one from my side. The physical Docker half is already recorded in the PR description and still holds: in a cgroup v2 memory-limited container an OOM-killed child exited 137 while the supervisor survived, host-side @cv — could you either point me at a host where I can run that, or record the waiver? Everything else on this PR is done and verified at |
|
Babysitting status for exact head b3ea62e (plain comment; no Changes Requested review from me): the earlier concrete findings have follow-up receipts and all 54 current check contexts are green. The remaining gate is freshness: this head is based on c4c020c while current main is da1b103, and maintainer edits are disabled. Please refresh the branch; I will re-run the exact-head gate after the new head is quiet for one hour. |
Sync fix/5796-host-side-oom-probe onto main da1b103 so the exact-head gate runs against current main. Maintainer edits are disabled on this PR, so the branch refresh is contributor-side. Signed-off-by: Shawn Xie <shaxie@nvidia.com>
|
Branch refreshed as requested. New exact head is Exact-head verification — after The architecture ratchet still holds on current main: One note for anyone refreshing this branch locally: main now carries The verification boundary in the PR description is unchanged: physical Docker evidence for the host-side counter remains valid but partial, and full |
|
Thanks for refreshing the branch. I learned after my request that conflict-free base refreshes are explicitly waived, so that request should not have been made solely for base currency. I will not ask for another such refresh; the new exact head is now protected by the one-hour quiet window while its CI/E2E evidence settles. This is a plain coordination comment, not Changes Requested. |
cjagwani
left a comment
There was a problem hiding this comment.
APPROVE for exact head ee41ccaf552a24891ec7c7c617eef19bfd85a178.
The earlier concrete findings are resolved: the CLI fixtures exercise labeled Docker discovery plus host-side docker exec, registry lookup failure maps to unavailable, and the test title follows the issue-suffix convention. I independently reproduced the focused probe suite (12/12), the text/JSON status suites (26/26), CLI build, and CLI typecheck. Required CI reports all 53 current checks green; CodeRabbit has no unresolved major/critical findings; all 10 commits are Verified; GitHub reports MERGEABLE.
Security review: PASS across the nine categories. The probe is read-only, uses an argv-style Docker invocation and a fixed shell program, requires a unique labeled container plus registry-backed longest-owner resolution, bounds execution to five seconds, parses only a numeric counter, exposes no secrets, and fails closed on unavailable or ambiguous state.
Scoped maintainer waiver: full live nemoclaw status evidence on a configured OpenShell Docker-driver host is waived for this patch. The recorded physical Docker experiment validates the supervisor-survives cgroup shape and per-container counter isolation; the focused integration contracts validate text/JSON degraded output, rebuild guidance, and nonzero exit; exact-head CI/E2E is green. This waiver is limited to this read-only transport change.
The deterministic gate's only non-pass is base age. Per the explicit conflict-free-refresh waiver, base currency alone is not a blocker while GitHub reports MERGEABLE; do not merge main again solely for freshness.
|
@cv The blocking review is attached to old head |
<!-- markdownlint-disable MD041 --> ## Summary Adds the canonical dated changelog entry for `v0.0.100` so the maintainer release plan can verify the pre-tag documentation prerequisite. The entry summarizes the user-facing changes merged since `v0.0.99` and links to the relevant guides. ## Changes - Add `docs/changelog/2026-07-31.mdx` with the exact `## v0.0.100` heading. - Cover restored OpenClaw pairing, transactional replacement, Deep Agents Code, onboarding recovery, lifecycle cleanup, Hermes builds, host provenance, documentation, and trusted E2E evidence. - Distinguish active Docker and Kubernetes runtime-bundle enforcement from the still-inactive managed shared-state transaction foundation. ## Source Coverage The release entry maps the doc-impacting merged PRs in the `v0.0.99..main` release range to `docs/changelog/2026-07-31.mdx`: #8021, #8024, #7973, #8028, #7947, #7788, #7884, #8023, #7969, #8020, #7989, #8000, #7907, #7942, #7567, #8013, #7955, #8017, #8014, #8015, #7629, #7644, #7821, #7971, and #7991. PR #7974 was reviewed after the final rebase and excluded because it changes internal maintainer-skill attribution policy and tests only; it does not change a user-facing product or documentation surface. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: the changelog contract test validates the dated entry, version heading, SPDX form, and route constraints. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `docs/changelog/2026-07-31.mdx`; exact-head review passed for `6093f44f`; writing rules and documentation style reviewed; `npx vitest run test/changelog-docs.test.ts` passed 6/6; `npm run docs` passed with zero Fern errors and two generic Fern upgrade notices. - Agent: Codex Desktop <!-- docs-review-head-sha: 6093f44 --> <!-- docs-review-agents-blob-sha: 3dd7c24 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable; no DGX Station host script changed. - Station profile/scenario: Not applicable. - Result: Not applicable. - Supporting evidence: Not applicable. ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run validate:pr` passed after refreshing `origin/main` when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: `npx vitest run test/changelog-docs.test.ts` passed 6/6 at `6093f44f`. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Not applicable to a dated prose-only release entry. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — validation passed with zero errors; Fern emitted two generic upgrade notices. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) — the changelog entry has the required parser-safe MDX SPDX header; dated changelog entries intentionally do not use page frontmatter. --- Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added release notes for v0.0.100. * Documented improvements to restore pairing, sandbox replacement, onboarding recovery, lifecycle cleanup, runtime handling, build support, host readiness, and end-to-end validation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Summary
nemoclaw statusreportedPhase: Readywith exit 0 and no rebuild hint after an OOM kill inside a terminal-runtime sandbox because the cgroup probe ran under the sandbox policy that denies/sys/fs/cgroup. This reads the sandbox container's counter through a bounded host-sidedocker exec, allowing the existing degraded-status contract to receive the signal it was designed to report.Related Issue
Fixes #5796
Changes
unavailablefor non-Docker drivers, registry failures, missing or ambiguous labeled containers, unresolved ownership, Docker execution failures, and malformed counter output.This replaces the ineffective in-sandbox probe transport. It does not add a fallback, compatibility layer, command, option, or configuration surface.
Type of Change
Quality Gates
b3ea62eb6; the lateree41ccaf5head is a clean main merge that leaves that reviewed diff byte-identical. All nine security categories pass with no findings; diff fingerprint352616bc3e775c44454432d5a7cd8b0e73111da5259e7242778dd5cf5b10ed67. The probe uses argv-style Docker execution with a fixed shell script, exact label filters, unique-container enforcement, registry-backed longest-owner validation, a five-second timeout, and fail-closed errors. It reads one counter and mutates no sandbox state.Documentation Writer Review
no-docs-neededdocs/reference/commands.mdxalready documents the terminal-runtime degraded OOM result, nonzero exit, structured JSON details, and rebuild guidance. Re-verified at headee41ccaf5: the main sync is a clean merge that leaves the PR diff at the same five files, touches no.md/.mdxordocs/path, and changes no user-visible surface. No documentation or code sample changed; no docs build was required.b3ea62eb6); Claude Code primary agent re-verified the unchanged scope atee41ccaf5Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpx vitest run --project cli src/lib/actions/sandbox/terminal-runtime-health.test.ts→ 12/12;npx vitest run --project integration test/cli/sandbox-status-text.test.ts test/cli/sandbox-status-json.test.ts→ 26/26;npx vitest run --project e2e-support test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts -t "carries the generated planner matrix"→ 1/1;npm run build:cli,npm run checks, andnpm run typecheck:clipassed.npm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result: not applicable; the change is scoped to one probe module, focused unit/CLI contracts, and the architecture ratchet it improves. Exact-head repository checks and normal commit/pre-push hooks passed.npm run docsbuilds without warnings (doc changes only)Verification boundary
The current exact head is
ee41ccaf5. It is signed, GitHub-Verified, and synced with mainda1b103121039fb2e056d41b901322f05489e479. Fresh exact-head CI/E2E and automated review are running.The physical Docker evidence remains valid but partial: in a cgroup v2 memory-limited container, an OOM-killed child exited 137 while the supervisor survived; host-side
docker execobservedoom_killchange from 0 to 1; a neighboring container remained at 0; and the shipped parser classified the victim asdegradedand the neighbor asok.Full
nemoclaw statusverification against a real OpenShell Docker-driver sandbox is still outstanding. The available maintainer host has OpenShell 0.0.85 but no configured gateway or registered sandbox. Approval therefore still requires exact-head OpenShell evidence showing the host counter, text and JSON degraded results, rebuild guidance, and nonzero exit, or an explicit maintainer waiver.Signed-off-by: Shawn Xie shaxie@nvidia.com
Summary by CodeRabbit