Skip to content

feat(state): segregate host state root per gateway port#6711

Open
laitingsheng wants to merge 36 commits into
mainfrom
feat/3053-instance-state-root
Open

feat(state): segregate host state root per gateway port#6711
laitingsheng wants to merge 36 commits into
mainfrom
feat/3053-instance-state-root

Conversation

@laitingsheng

@laitingsheng laitingsheng commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

A non-default NEMOCLAW_GATEWAY_PORT previously shared the single ~/.nemoclaw/ host state root, so its sandbox registry, legacy credential-migration files, snapshots, and onboarding state collided with the default gateway's. This routes the host state root through one nemoclawStateRoot resolver keyed off the existing gateway port, so a non-default port now owns a separate ~/.nemoclaw/gateways/<port>/ root while the default port stays byte-identical at ~/.nemoclaw/.

Related Issue

Fixes #3053

Changes

  • Add src/lib/state/state-root.tsnemoclawStateRoot(home, gatewayPort) returns ~/.nemoclaw for the default gateway port and ~/.nemoclaw/gateways/<port> otherwise. It is keyed off the existing GATEWAY_PORT constant so the state root and the gateway name (nemoclaw / nemoclaw-<port>) always agree.
  • Route the host state root through the resolver at every writer: the sandbox registry (sandboxes.json), legacy credential-migration path, config-io permission-heal and remediation, onboard session, rebuild backups, blueprint cache, model router, local-inference adapter state and Ollama proxy token, share mounts, usage notice, and onboard-failure diagnostics.
  • Keep the CLI source clone and the managed_swap marker at the shared ~/.nemoclaw/ root — one CLI binary and one host swap file exist per machine, not per gateway.
  • Uninstall now targets the port-scoped state root, always preserves the gateways/ subtree so other environments survive even --destroy-user-data, prints the resolved state directory in its plan output, and skips host-swap teardown while other gateway-port environments remain.
  • Document the per-port state root in docs/manage-sandboxes/lifecycle.mdx and docs/reference/architecture.mdx.

This adds one resolver abstraction because the ~/.nemoclaw root was inlined at every state-path site with no single lever; segregation needs a single keyed source of truth. Its current consumer is #3053 (two segregated NemoClaw environments on one host). The default-port byte-identical contract is protected by src/lib/state/state-root.test.ts; the end-to-end per-port fork by src/lib/state/registry-port-segregation.test.ts; and the uninstall preservation plus host-swap guard by src/lib/actions/uninstall/run-plan-gateway-segregation.test.ts.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: mechanical state-root path routing that preserves byte-identical default-port behavior (test-covered); no change to credential handling, auth, or policy semantics. Requesting maintainer sensitive-path review.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result: npm run test:changed — 154 files and 1,541 tests passed; npm run checks — passed; npm run typecheck:cli — 0 errors
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Tinson Lai tinsonl@nvidia.com

Summary by CodeRabbit

  • New Features
    • Non-default gateway ports now isolate host state under ~/.nemoclaw/gateways/<port>/ (default remains in ~/.nemoclaw/).
    • uninstall adds --gateway <name> and applies port-scoped targeting/cleanup.
    • Dashboard port allocation avoids collisions across multiple gateways on the same host.
  • Bug Fixes
    • Uninstall cleanup is properly scoped, preserving other ports’ registries/state and shared swap where needed.
    • Port-scoped state/config writes and diagnostics now follow correct locations with stronger symlink safety.
  • Documentation
    • Updated lifecycle and uninstall guidance for port-scoped preservation and --gateway behavior.
  • Tests
    • Expanded coverage for port segregation, uninstall scoping (including mismatch failure), symlink-safe state writers, and cross-gateway dashboard port allocation.

A non-default NEMOCLAW_GATEWAY_PORT now resolves its own host state
root at ~/.nemoclaw/gateways/<port>/, covering the sandbox registry,
credentials, snapshots, onboard session, and inference adapter state,
so two gateways on one host stay fully segregated. The default port
keeps the shared ~/.nemoclaw/ location, so existing single-gateway
hosts are byte-identical.

Route the registry, credentials, config-io, onboard, inference,
share, and diagnostics paths through one nemoclawStateRoot resolver
keyed off GATEWAY_PORT. Keep the CLI source clone and the managed_swap
marker host-shared. Uninstall targets the port-scoped state root,
always preserves the gateways/ subtree so other environments survive
even --destroy-user-data, and no longer tears down the host-shared
swap file while other gateway-port environments remain.

Fixes #3053

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2b94a515-95ec-4f88-b3f7-c3f5cb7db73c

📥 Commits

Reviewing files that changed from the base of the PR and between 09682cc and 290da9b.

📒 Files selected for processing (2)
  • docs/reference/commands.mdx
  • test/onboard-model-router.test.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/onboard-model-router.test.ts
  • docs/reference/commands.mdx

📝 Walkthrough

Walkthrough

The change centralizes gateway-aware state paths, migrates legacy state, scopes uninstall and installer operations, adds registry and symlink-safety validation, and documents non-default gateway lifecycle behavior.

Changes

Gateway-port state segregation

Layer / File(s) Summary
State-root contract and consumers
src/lib/state/*, src/lib/credentials/*, src/lib/inference/*, src/lib/onboard/*, src/lib/share-command.ts, src/lib/domain/uninstall/paths.ts
Non-default ports use ~/.nemoclaw/gateways/<port>/ for gateway-specific state while default-port behavior remains under ~/.nemoclaw/.
Legacy migration and host registry
src/lib/state/gateway-registry.ts, src/lib/state/legacy-port-migration.ts, src/lib/cli/public-dispatch.ts, scripts/install.sh
Legacy state is validated and migrated when gateway ownership is provable; ambiguous state remains in place.
Scoped uninstall preservation
src/lib/actions/uninstall/run-plan.ts, src/commands/internal/uninstall/run-plan.ts, src/lib/actions/uninstall/openrouter-runtime-adapter-cleanup.ts
Uninstall targets the port-selected gateway, removes selected resources, and preserves sibling gateways and shared host resources.
Isolation and security validation
src/**/*.test.ts, test/**/*.test.ts
Tests cover registry segregation, migration, installer recovery, dashboard-port collisions, concurrent gateways, selected-root credentials, and symlink-safe writes.
Gateway-port documentation
docs/manage-sandboxes/lifecycle.mdx, docs/reference/architecture.mdx, docs/reference/commands.mdx
Documentation describes port-scoped state, migration rules, provider credential storage, and uninstall preservation behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant migrateLegacyPortState
  participant GatewayRegistry
  participant SelectedGateway
  participant SharedState
  CLI->>migrateLegacyPortState: migrate selected port state
  migrateLegacyPortState->>GatewayRegistry: inspect registry identities
  GatewayRegistry->>SelectedGateway: write owned registry and state
  migrateLegacyPortState->>SharedState: retain ambiguous or remaining state
  CLI->>SelectedGateway: continue selected-port command
Loading

Possibly related PRs

Suggested labels: area: cli

Suggested reviewers: cjagwani

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 13.10% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely captures the core change: host state is now segregated by gateway port.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/3053-instance-state-root

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the branch.

TypeScript / code-coverage/cli

The overall coverage in the branch remains at 79%, unchanged from the branch.

Show a code coverage summary of the most impacted files.
File b85f5b2 290da9b +/-
src/lib/state/registry.ts 90% 85% -5%
src/lib/inference/local.ts 71% 68% -3%
src/lib/state/o...oard-session.ts 85% 84% -1%
src/lib/actions...all/run-plan.ts 81% 83% +2%
src/lib/onboard...usage-notice.ts 22% 30% +8%
src/lib/onboard...eate-failure.ts 79% 90% +11%
src/lib/inferen...er-lifecycle.ts 57% 70% +13%
src/lib/state/state-root.ts 0% 83% +83%
src/lib/state/l...rt-migration.ts 0% 90% +90%
src/lib/state/g...way-registry.ts 0% 95% +95%

Updated July 14, 2026 05:24 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

Copy link
Copy Markdown
Contributor

@laitingsheng laitingsheng added area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery feature PR adds or expands user-visible functionality labels Jul 12, 2026
@github-actions

github-actions Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Primary next action: No advisor follow-up required beyond maintainer review.
Findings: 0 blockers · 0 warnings · 0 optional suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (non-blocking second opinion): Failed after a partial review · low confidence · 1 blocker · 11 warnings · 1 suggestion

Nemotron is a non-blocking second opinion. Its prose, findings, and E2E guidance do not change the primary assessment above and remain in workflow artifacts only.

Since last review: 0 prior items resolved · 0 still apply · 0 new items found

E2E guidance

Advisory only: coverage and selector recommendations are non-authoritative. E2E / PR Gate independently computes and dispatches trusted jobs without consuming this output.

Recommended coverage: cloud-onboard, credential-sanitization, security-posture, concurrent-gateway-ports, inference-routing, network-policy, onboard-repair, onboard-resume
Recommended selectors: cloud-onboard, credential-sanitization, security-posture, concurrent-gateway-ports, inference-routing, network-policy, onboard-repair, onboard-resume

  • cloud-onboard — Selected from the trusted checked-in E2E coverage inventory.

  • credential-sanitization — Selected from the trusted checked-in E2E coverage inventory.

  • security-posture — Selected from the trusted checked-in E2E coverage inventory.

  • concurrent-gateway-ports — Selected from the trusted checked-in E2E coverage inventory.

  • inference-routing — Selected from the trusted checked-in E2E coverage inventory.

  • network-policy — Selected from the trusted checked-in E2E coverage inventory.

  • onboard-repair — Selected from the trusted checked-in E2E coverage inventory.

  • onboard-resume — Selected from the trusted checked-in E2E coverage inventory.

  • cloud-onboard — Selected as a trusted checked-in E2E job.

  • credential-sanitization — Selected as a trusted checked-in E2E job.

  • security-posture — Selected as a trusted checked-in E2E job.

  • concurrent-gateway-ports — Selected as a trusted checked-in E2E job.

  • inference-routing — Selected as a trusted checked-in E2E job.

  • network-policy — Selected as a trusted checked-in E2E job.

  • onboard-repair — Selected as a trusted checked-in E2E job.

  • onboard-resume — Selected as a trusted checked-in E2E job.

Workflow run details

This is an automated, non-authoritative review. Findings are inputs to maintainer adjudication. Warnings and optional suggestions do not require a response or follow-up. A human maintainer makes the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/manage-sandboxes/lifecycle.mdx`:
- Around line 154-157: Update the agent-specific lifecycle documentation block
to replace the invalid $$nemoclaw uninstall command with the literal Deep Agents
CLI command nemo-deepagents uninstall, while preserving the surrounding
gateway-state behavior and wording.
- Line 150: Update the lifecycle documentation near the NEMOCLAW_GATEWAY_PORT
state-root description to replace “fully segregated” with a narrower claim about
gateway-scoped state. Mention that the CLI source clone and managed_swap marker
remain shared under ~/.nemoclaw/ if describing shared host-level artifacts,
while preserving the separate registry, credentials, snapshots, and uninstall
behavior.

In `@docs/reference/architecture.mdx`:
- Line 311: Update the preceding ~/.nemoclaw/sandboxes.json documentation row to
state explicitly that it is the registry for the default gateway port only;
retain the separate non-default registry location described by the
gateways/<port> entry.

In `@src/lib/actions/uninstall/run-plan-gateway-segregation.test.ts`:
- Around line 70-73: Remove the conditional inside the mock run handler and
record every invoked command unconditionally in swapCommands. Update the related
assertions to verify unwanted commands with not.toContain, preserving the test’s
intent without adding an if statement.

In `@src/lib/onboard/usage-notice.ts`:
- Line 56: Update the usage-notice path construction to pass the active gateway
port to nemoclawStateRoot instead of relying on its default-port behavior.
Preserve the existing HOME fallback and usage-notice.json filename while
ensuring non-default gateways resolve their port-scoped state root.

In `@src/lib/share-command.ts`:
- Line 64: The defaultShareMountDir resolver currently ignores gateway identity,
causing all gateways to use the same mount root. Update defaultShareMountDir to
accept the active gateway port and include that port when deriving the mount
directory, then update every caller to pass the corresponding gateway port so
non-default gateways use isolated mount state.

In `@src/lib/state/onboard-session.ts`:
- Around line 48-50: Update the onboarding session path definitions around
SESSION_DIR, SESSION_FILE, and LOCK_FILE to derive the directory using the
active gateway port, rather than a single module-global nemoclawStateRoot value.
Ensure each non-default gateway receives distinct session and lock paths while
preserving the existing filenames.

In `@src/lib/state/state-root.test.ts`:
- Around line 37-41: Update the getNemoclawStateRoot default-port test to
explicitly control NEMOCLAW_GATEWAY_PORT, clearing it before module
initialization or using the explicit-port helper to assert default behavior.
Ensure the test remains isolated from the host environment and preserves the
expected shared root assertion.

In `@src/lib/state/state-root.ts`:
- Around line 19-24: Update resolveHome to use an empty-value fallback, matching
the credentials store behavior, so empty HOME values fall back to os.homedir()
before getNemoclawStateRoot constructs the state path. Keep the existing
getNemoclawStateRoot and nemoclawStateRoot flow unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3fa4a17d-c76f-46ab-82a5-a14de9de2fd0

📥 Commits

Reviewing files that changed from the base of the PR and between e04a344 and 8d028b8.

📒 Files selected for processing (21)
  • docs/manage-sandboxes/lifecycle.mdx
  • docs/reference/architecture.mdx
  • src/lib/actions/uninstall/run-plan-gateway-segregation.test.ts
  • src/lib/actions/uninstall/run-plan.ts
  • src/lib/credentials/store.ts
  • src/lib/domain/uninstall/paths.ts
  • src/lib/inference/local-adapter-lifecycle.ts
  • src/lib/inference/local.ts
  • src/lib/onboard/docker-gpu-patch.ts
  • src/lib/onboard/model-router.ts
  • src/lib/onboard/sandbox-create-failure.ts
  • src/lib/onboard/usage-notice.ts
  • src/lib/share-command.ts
  • src/lib/state/config-io.ts
  • src/lib/state/onboard-session.ts
  • src/lib/state/paths.ts
  • src/lib/state/registry-port-segregation.test.ts
  • src/lib/state/registry.ts
  • src/lib/state/sandbox.ts
  • src/lib/state/state-root.test.ts
  • src/lib/state/state-root.ts

Comment thread docs/manage-sandboxes/lifecycle.mdx Outdated
Comment thread docs/manage-sandboxes/lifecycle.mdx Outdated
Comment thread docs/reference/architecture.mdx Outdated
Comment thread src/lib/actions/uninstall/run-plan-gateway-segregation.test.ts
Comment thread src/lib/onboard/usage-notice.ts Outdated
Comment thread src/lib/share-command.ts Outdated
Comment thread src/lib/state/onboard-session.ts Outdated
Comment thread src/lib/state/state-root.test.ts
Comment thread src/lib/state/state-root.ts
Fixes a regression the port-keyed state root introduced in a pre-existing
CLI status test that temporarily overrides NEMOCLAW_GATEWAY_PORT to probe
for a gateway-port conflict, closes the empty-HOME relative-path edge
case in the state-root resolver, makes the default-port unit test
control the environment variable it claims to test, corrects two doc
wording issues (an overstated segregation claim and a Deep Agents
command name), and drops a conditional from the new uninstall test to
satisfy the codebase-growth guardrail.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/repro-2666-silent-list-status.test.ts`:
- Line 210: Strengthen the port-specific scenario setup around seedRegistry so
the default ~/.nemoclaw registry cannot satisfy the test: clear it before the
scenario or seed it with distinct sentinel data, then assert the selected result
comes only from nemoclawStateRoot(home, port). Apply the same correction to the
corresponding occurrence.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 54a56463-3080-4275-8500-99fb477832c9

📥 Commits

Reviewing files that changed from the base of the PR and between 8d028b8 and d560c34.

📒 Files selected for processing (6)
  • docs/manage-sandboxes/lifecycle.mdx
  • docs/reference/architecture.mdx
  • src/lib/actions/uninstall/run-plan-gateway-segregation.test.ts
  • src/lib/state/state-root.test.ts
  • src/lib/state/state-root.ts
  • test/repro-2666-silent-list-status.test.ts
🚧 Files skipped from review as they are similar to previous changes (5)
  • src/lib/state/state-root.ts
  • src/lib/state/state-root.test.ts
  • docs/manage-sandboxes/lifecycle.mdx
  • src/lib/actions/uninstall/run-plan-gateway-segregation.test.ts
  • docs/reference/architecture.mdx

Comment thread test/repro-2666-silent-list-status.test.ts
The port-specific #3271 scenario seeded identical my-assist data under
both the default and port-scoped registries, so a regression that kept
reading the default root under a non-default NEMOCLAW_GATEWAY_PORT
would still pass. Clear the default registry before that scenario so
the test only succeeds when the CLI actually reads the port-scoped
root.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
@cv cv added v0.0.82 Release target and removed v0.0.82 Release target labels Jul 12, 2026
…ate-root

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

# Conflicts:
#	src/lib/onboard/docker-gpu-patch.ts
@cv cv added the v0.0.82 Release target label Jul 13, 2026
Every production call site relied on nemoclawStateRoot's default
gatewayPort parameter, which resolves to the correct value only
because GATEWAY_PORT happens to be a process-wide constant computed
once at module load. That's easy to misread as always resolving the
default port. Thread GATEWAY_PORT through all 18 call sites explicitly
so each one is self-evidently correct without tracing into
core/ports.ts. No behavioural change — GATEWAY_PORT is the exact value
the default parameter already supplied.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
@prekshivyas prekshivyas self-assigned this Jul 13, 2026
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

E2E Target Results — ⚠️ Some tests cancelled — partial pass

Run: 29265242441
Workflow ref: feat/3053-instance-state-root
Requested targets: (default — all supported)
Requested test IDs: credential-sanitization,security-posture,inference-routing,network-policy,onboard-repair,onboard-resume,cloud-onboard,concurrent-gateway-ports
Summary: 1 passed, 0 failed, 7 cancelled, 0 skipped, 0 unknown

Test Result
cloud-onboard ⚠️ cancelled
concurrent-gateway-ports ⚠️ cancelled
credential-sanitization ⚠️ cancelled
inference-routing ✅ success
network-policy ⚠️ cancelled
onboard-repair ⚠️ cancelled
onboard-resume ⚠️ cancelled
security-posture ⚠️ cancelled

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

E2E Target Results — ❌ Some tests failed

Run: 29265796896
Workflow ref: feat/3053-instance-state-root
Requested targets: (default — all supported)
Requested test IDs: credential-sanitization,security-posture,inference-routing,network-policy,onboard-repair,onboard-resume,concurrent-gateway-ports,cloud-onboard,model-router-provider-routed-inference
Summary: 4 passed, 1 failed, 4 cancelled, 0 skipped, 0 unknown

Test Result
cloud-onboard ⚠️ cancelled
concurrent-gateway-ports ❌ failure
credential-sanitization ✅ success
inference-routing ✅ success
model-router-provider-routed-inference ⚠️ cancelled
network-policy ⚠️ cancelled
onboard-repair ✅ success
onboard-resume ✅ success
security-posture ⚠️ cancelled

Failed tests: concurrent-gateway-ports. Check run artifacts for logs.

@github-actions

Copy link
Copy Markdown
Contributor

E2E Target Results — ❌ Some tests failed

Run: 29266737670
Workflow ref: feat/3053-instance-state-root
Requested targets: (default — all supported)
Requested test IDs: credential-sanitization,security-posture,inference-routing,network-policy,onboard-repair,onboard-resume,concurrent-gateway-ports,cloud-onboard,model-router-provider-routed-inference
Summary: 8 passed, 1 failed, 0 cancelled, 0 skipped, 0 unknown

Test Result
cloud-onboard ✅ success
concurrent-gateway-ports ❌ failure
credential-sanitization ✅ success
inference-routing ✅ success
model-router-provider-routed-inference ✅ success
network-policy ✅ success
onboard-repair ✅ success
onboard-resume ✅ success
security-posture ✅ success

Failed tests: concurrent-gateway-ports. Check run artifacts for logs.

@cjagwani

Copy link
Copy Markdown
Contributor

Exact-head live-E2E diagnosis for the failing concurrent-gateway-ports target: the failure at test/e2e/live/concurrent-gateway-ports.test.ts:424 is consistent with this PR's intended state-root segregation, not a transient runner error. After sandbox B is onboarded under NEMOCLAW_GATEWAY_PORT=, the test invokes a single unscoped nemoclaw list (default gateway A) and still expects both A and B. With per-gateway sandboxes.json, that command correctly sees only A. Please update the live proof to list each instance with its own NEMOCLAW_GATEWAY_PORT, assert A is present only in A's registry and B only in B's, and derive each dashboard port from the corresponding scoped output. Then rerun the exact-head required live targets. This closes the detection gap while matching #3053's stated requirement that each instance own its own registry/state root.

cv pushed a commit that referenced this pull request Jul 13, 2026
## Summary

- validate the pinned Ubuntu `fd-find` binary using its actual `fdfind
<version>` identity
- keep the trusted workflow-boundary validator aligned with the
executable check
- make the workflow test fixture reproduce the package output seen on
hosted runners

## Why

PR review advisors currently fail during SDK bootstrap before analysis.
The pinned Ubuntu package reports `fdfind 9.0.0`, while the workflow
expects `fd 9.0.0`. This was reproduced on both advisor lanes for #6711.

## Validation

- `npx vitest run --project integration
test/pr-review-advisor-workflow-boundary.test.ts` (19 passed)
- `npm run checks`
- pre-commit and pre-push hooks, including CLI typecheck


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Corrected verification of the `fdfind` search tool version during Pi
SDK installation.
* Improved workflow boundary validation to align with the tool’s actual
version output and mismatch messaging.
* Ensured YAML is explicitly pinned and validated during Pi SDK
installation and related workflow jobs, improving consistency and
reliability of the PR review automation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@github-actions

Copy link
Copy Markdown
Contributor

E2E Target Results — ⚠️ Some tests cancelled — partial pass

Run: 29286659660
Workflow ref: feat/3053-instance-state-root
Requested targets: (default — all supported)
Requested test IDs: cloud-onboard,credential-sanitization,security-posture,inference-routing,network-policy,onboard-repair,onboard-resume,concurrent-gateway-ports
Summary: 1 passed, 0 failed, 7 cancelled, 0 skipped, 0 unknown

Test Result
cloud-onboard ⚠️ cancelled
concurrent-gateway-ports ⚠️ cancelled
credential-sanitization ⚠️ cancelled
inference-routing ✅ success
network-policy ⚠️ cancelled
onboard-repair ⚠️ cancelled
onboard-resume ⚠️ cancelled
security-posture ⚠️ cancelled

@prekshivyas prekshivyas marked this pull request as draft July 13, 2026 21:56
@copy-pr-bot

copy-pr-bot Bot commented Jul 13, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@prekshivyas prekshivyas marked this pull request as ready for review July 13, 2026 21:56
@github-actions

Copy link
Copy Markdown
Contributor

E2E Target Results — ✅ All requested tests passed

Run: 29287712163
Workflow ref: feat/3053-instance-state-root
Requested targets: (default — all supported)
Requested test IDs: cloud-onboard,credential-sanitization,security-posture,inference-routing,network-policy,onboard-repair,onboard-resume,concurrent-gateway-ports
Summary: 8 passed, 0 failed, 0 cancelled, 0 skipped, 0 unknown

Test Result
cloud-onboard ✅ success
concurrent-gateway-ports ✅ success
credential-sanitization ✅ success
inference-routing ✅ success
network-policy ✅ success
onboard-repair ✅ success
onboard-resume ✅ success
security-posture ✅ success

@prekshivyas

Copy link
Copy Markdown
Contributor

Exact-head maintainer handoff for 3c263c953f7c01b54568166509db1c4de3598228 on base 989eea9ac796537b9a803967ee1aed68dc51dec0:

  • Full non-secret CI is green, including all 8 CLI shards, aggregate checks, installer integration, self-hosted isolation tests, macOS/WSL, CodeQL, ShellCheck, and docs checks: run 29289384566.
  • Both exact-head PR Advisor jobs are green. The canonical Terra ledger reports 0 blockers, 0 warnings, and 0 suggestions: run 29289383368.
  • CodeRabbit is green and all review threads are resolved (0 unresolved).
  • The four lifecycle/upgrade issues from the human change requests are implemented and covered: selected-only uninstall with legacy siblings, strict nondefault migration, host-wide dashboard collision detection, and conservative default/shared sibling detection.
  • The selected live E2E set passed 8/8 at product revision e66655cb3251e0a010b8c5732603129b37e07c33: run 29287712163. The PR-controlled concurrent-gateway-ports test is byte-identical between that passing revision and the current head.

The remaining red E2E / PR Gate is the expected no-secret failure because this PR changes a trusted E2E test. It requires a repository maintainer/admin to run E2E / PR Gate on main with:

  • operation: resolve-control-plane
  • PR: 6711
  • expected head: 3c263c953f7c01b54568166509db1c4de3598228
  • expected base: 989eea9ac796537b9a803967ee1aed68dc51dec0
  • evidence: https://github.com/NVIDIA/NemoClaw/actions/runs/29287712163
  • reason: Independent review confirms the PR-controlled concurrent-gateway test is byte-identical to revision e66655cb, where all eight selected credentialed targets passed in run 29287712163. The current head only adds uninstall-selector docs/contract coverage and merges trusted main gate changes; no PR-controlled code will be run with repository credentials.

@cjagwani please re-review the current head and, if satisfied with the independent E2E evidence, record the exact-diff control-plane exception. No merge action has been taken.

# Conflicts:
#	src/lib/onboard/dashboard-port.test.ts
#	src/lib/onboard/dashboard-port.ts
@prekshivyas

Copy link
Copy Markdown
Contributor

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (4)
src/lib/actions/uninstall/run-plan.ts (1)

1170-1205: 📐 Maintainability & Code Quality | 🔵 Trivial

Redundant re-computation of gateway-sibling inspection.

removeManagedSwap recomputes inspectOtherGatewayEnvironments(paths, runtime) from scratch (line 1182), even though executePlan is already called with the equivalent scopedToSelectedGateway (computed once in runUninstallPlan, line 1463). This duplicates a non-trivial filesystem walk + registry read and creates two copies of the same "other gateway remains" logic that could silently drift apart.

♻️ Thread the already-computed result through instead of recomputing
-function removeManagedSwap(paths: UninstallPaths, runtime: UninstallRuntime): void {
+function removeManagedSwap(
+  paths: UninstallPaths,
+  runtime: UninstallRuntime,
+  otherGatewayEnvironmentsRemain: boolean,
+): void {
   ...
-  if (inspectOtherGatewayEnvironments(paths, runtime).otherGatewayEnvironmentsRemain) {
+  if (otherGatewayEnvironmentsRemain) {
     runtime.log(
       "Other NemoClaw gateway-port environments remain; keeping the host-shared /swapfile.",
     );
     return;
   }

And update the call site in executePlan to pass scopedToSelectedGateway through.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/uninstall/run-plan.ts` around lines 1170 - 1205, Update
removeManagedSwap to accept the already-computed scopedToSelectedGateway result
instead of calling inspectOtherGatewayEnvironments. Use that passed result for
the other-gateway-environments check, and update executePlan to forward
scopedToSelectedGateway when invoking removeManagedSwap; leave the existing
cleanup behavior unchanged.
src/lib/state/legacy-port-migration.ts (2)

361-367: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Clarify ambiguous-looking precedence in the ownership invariant check.

rawMetadata.rewriteLegacyRegistry !== selectedSandboxNames.length > 0 is correct (> binds tighter than !==), but reads as if !== applies to selectedSandboxNames.length alone. This guards a resumable-migration consistency check, so misreading it during a future edit is a real risk.

✏️ Proposed clarification
-    rawMetadata.rewriteLegacyRegistry !== selectedSandboxNames.length > 0 ||
+    rawMetadata.rewriteLegacyRegistry !== (selectedSandboxNames.length > 0) ||
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/legacy-port-migration.ts` around lines 361 - 367, Clarify the
first ownership invariant condition in the migration validation check by
explicitly grouping the selectedSandboxNames.length > 0 boolean expression
before comparing it with rawMetadata.rewriteLegacyRegistry. Preserve the
existing logic and leave the other consistency conditions unchanged.

448-470: 🩺 Stability & Availability | 🔵 Trivial

Crash windows leave orphaned .preparing.*/.completed.* directories.

If the process crashes between the atomic rename and the subsequent fs.rmSync cleanup in createMigrationIntent (line 469) or removeMigrationIntent (line 482), the renamed-away directory is never swept on a later run — readMigrationIntent only looks for the literal MIGRATION_INTENT name, so these won't corrupt state, but they will accumulate as disk debt over repeated crashes. Consider a startup sweep for stale ${MIGRATION_INTENT}.preparing.*/${MIGRATION_INTENT}.completed.* siblings.

Also applies to: 477-484

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/legacy-port-migration.ts` around lines 448 - 470, Add startup
cleanup for stale siblings matching the migration intent’s
`${MIGRATION_INTENT}.preparing.*` and `${MIGRATION_INTENT}.completed.*` naming
patterns, removing them before normal intent processing. Reuse the existing
safe-path validation and filesystem conventions used by `createMigrationIntent`,
`removeMigrationIntent`, and `readMigrationIntent`; preserve the active literal
`MIGRATION_INTENT` directory and avoid deleting unrelated entries.
test/install-gateway-state-root.test.ts (1)

40-61: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Sanitize inherited NEMOCLAW_* env vars for determinism.

This spawns with the full parent process.env and only blanks a fixed allowlist of port vars. test/uninstall.test.ts already establishes a sanitizedParentEnv() pattern that strips all NEMOCLAW_* keys before setting the ones under test; using the same approach here would avoid leaking unrelated inherited NEMOCLAW_* config (e.g., from CI or a dev shell) into these gateway-state-root assertions.

♻️ Proposed sanitization
+function sanitizedParentEnv(): NodeJS.ProcessEnv {
+  return Object.fromEntries(
+    Object.entries(process.env).filter(([key]) => !key.startsWith("NEMOCLAW_")),
+  ) as NodeJS.ProcessEnv;
+}
+
 function runInstallerFunctions(
   home: string,
   body: string,
 ): { output: string; status: number | null } {
   const result = spawnSync("bash", ["-c", `source "${INSTALLER}" >/dev/null\n${body}`], {
     encoding: "utf8",
     env: {
-      ...process.env,
+      ...sanitizedParentEnv(),
       BASH_ENV: "",
       ENV: "",
       HOME: home,
-      NEMOCLAW_BEDROCK_RUNTIME_ADAPTER_PORT: "",
-      NEMOCLAW_DASHBOARD_PORT: "",
       NEMOCLAW_GATEWAY_PORT: "9123",
-      NEMOCLAW_OLLAMA_PORT: "",
-      NEMOCLAW_OLLAMA_PROXY_PORT: "",
-      NEMOCLAW_OPENROUTER_RUNTIME_ADAPTER_PORT: "",
-      NEMOCLAW_VLLM_PORT: "",
     },
   });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/install-gateway-state-root.test.ts` around lines 40 - 61, Update
runInstallerFunctions to build its child-process environment from
sanitizedParentEnv(), removing all inherited NEMOCLAW_* variables before
applying HOME and the specific port values required by these assertions. Reuse
the existing sanitization pattern from test/uninstall.test.ts rather than
maintaining a fixed list of variables.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/commands.mdx`:
- Around line 3045-3056: In the “User-data preservation under ~/.nemoclaw/”
section, split each multi-sentence Markdown/MDX line into separate lines so
every period-terminated sentence occupies its own source line, preserving the
existing wording and formatting.

---

Nitpick comments:
In `@src/lib/actions/uninstall/run-plan.ts`:
- Around line 1170-1205: Update removeManagedSwap to accept the already-computed
scopedToSelectedGateway result instead of calling
inspectOtherGatewayEnvironments. Use that passed result for the
other-gateway-environments check, and update executePlan to forward
scopedToSelectedGateway when invoking removeManagedSwap; leave the existing
cleanup behavior unchanged.

In `@src/lib/state/legacy-port-migration.ts`:
- Around line 361-367: Clarify the first ownership invariant condition in the
migration validation check by explicitly grouping the
selectedSandboxNames.length > 0 boolean expression before comparing it with
rawMetadata.rewriteLegacyRegistry. Preserve the existing logic and leave the
other consistency conditions unchanged.
- Around line 448-470: Add startup cleanup for stale siblings matching the
migration intent’s `${MIGRATION_INTENT}.preparing.*` and
`${MIGRATION_INTENT}.completed.*` naming patterns, removing them before normal
intent processing. Reuse the existing safe-path validation and filesystem
conventions used by `createMigrationIntent`, `removeMigrationIntent`, and
`readMigrationIntent`; preserve the active literal `MIGRATION_INTENT` directory
and avoid deleting unrelated entries.

In `@test/install-gateway-state-root.test.ts`:
- Around line 40-61: Update runInstallerFunctions to build its child-process
environment from sanitizedParentEnv(), removing all inherited NEMOCLAW_*
variables before applying HOME and the specific port values required by these
assertions. Reuse the existing sanitization pattern from test/uninstall.test.ts
rather than maintaining a fixed list of variables.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7c61d79b-093b-4179-83d3-45c0bf3b8e69

📥 Commits

Reviewing files that changed from the base of the PR and between 3c5fb29 and 8a672bd.

📒 Files selected for processing (18)
  • docs/manage-sandboxes/lifecycle.mdx
  • docs/reference/architecture.mdx
  • docs/reference/commands.mdx
  • scripts/install.sh
  • src/lib/actions/uninstall/run-plan-gateway-segregation.test.ts
  • src/lib/actions/uninstall/run-plan-preserved-registry.test.ts
  • src/lib/actions/uninstall/run-plan.test.ts
  • src/lib/actions/uninstall/run-plan.ts
  • src/lib/onboard/dashboard-port.test.ts
  • src/lib/onboard/dashboard-port.ts
  • src/lib/state/legacy-port-migration.test.ts
  • src/lib/state/legacy-port-migration.ts
  • test/fixtures/uninstall-prompt-pty-driver.ts
  • test/install-gateway-state-root.test.ts
  • test/install-openshell-upgrade-prompt.test.ts
  • test/install-preexisting-sandbox-recovery.test.ts
  • test/mcp-lifecycle-lock.test.ts
  • test/uninstall.test.ts
🚧 Files skipped from review as they are similar to previous changes (5)
  • src/lib/actions/uninstall/run-plan.test.ts
  • docs/manage-sandboxes/lifecycle.mdx
  • src/lib/onboard/dashboard-port.ts
  • docs/reference/architecture.mdx
  • scripts/install.sh

Comment thread docs/reference/commands.mdx
@prekshivyas

Copy link
Copy Markdown
Contributor

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@prekshivyas

Copy link
Copy Markdown
Contributor

@coderabbitai resume

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Reviews resumed.

@prekshivyas

Copy link
Copy Markdown
Contributor

@cjagwani ready for exact-head re-review at ada776f52efa136726dde258fc72a79f667ca9ca.

Your latest shared-registry uninstall blocker is fixed: every valid legacy row not owned by the selected gateway now keeps cleanup scoped, including two non-default legacy rows after migrating one. The plan-level regression proves no sandbox delete --all, no provider deletion, and sibling registry/runtime preservation. The earlier uninstall isolation, migration, dashboard-collision, and default-registry/swap boundaries remain covered. Recovery commands now bypass a broken migration gate, while ordinary stateful commands still fail closed; stale exact-name migration intents are cleaned under the migration lock with symlink refusal.

Exact-head regular CI is green, CodeRabbit has no unresolved threads, and primary Advisor is 0 blockers / 0 warnings / 0 suggestions. Trusted E2E dispatch is currently blocked by the base-branch controller crash tracked in #6823; both failing gate jobs execute the broken main script rather than this PR code.

cv pushed a commit that referenced this pull request Jul 14, 2026
## Summary
- replace the unsupported TypeScript constructor parameter property in
the PR E2E gate
- add a regression that launches the controller with the exact `node
--experimental-strip-types` runtime used by Actions

## Why
Current `main` crashes every PR E2E gate during initialization with
`ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX`, before it can evaluate the PR. This
is visible on #6770, #6711, and #6672.

## Validation
- `npx vitest run --project integration test/pr-e2e-gate-command.test.ts
test/pr-e2e-gate-lifecycle.test.ts test/pr-e2e-gate-exceptions.test.ts
test/pr-e2e-gate.test.ts test/pr-e2e-gate-workflow.test.ts` (108 passed)
- `npm run build:cli`
- pre-commit and pre-push hooks

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved validation for invalid end-to-end gate modes, including
clearer error messaging and the correct failure status.
* Prevented unsupported TypeScript syntax errors when running the gate
command directly in Node’s experimental type-stripping mode.


<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@prekshivyas

Copy link
Copy Markdown
Contributor

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@prekshivyas

Copy link
Copy Markdown
Contributor

Exact-head re-review request for 09682cc53db45600bb64085a07d76808f1bd0df2 (base 7b1382924d29097e96aeecf34505c2e247d12c8d).

The two prior uninstall reviews remain covered: selected-port uninstall preserves all sibling/shared resources, and every valid legacy row not owned by the selected gateway counts as a sibling before global cleanup. The fresh Advisor blocker is also fixed: supported internal uninstall plan/run-plan now performs legacy port migration first and fails closed before destructive execution when migration is unsafe; focused dispatcher/uninstall regressions cover both paths.

Full check:diff, exact-head CI (including all eight shards), canonical GPT Advisor 0/0/0, CodeRabbit, and all review threads are green. The required eight-target E2E set also passed on the immediately preceding branch revision in run 29287712163.

The remaining gate is the expected exact-head control-plane exception because this PR changes trusted E2E code. Gate run 29305428800 names the exact head/base and selected jobs. @cjagwani, please re-review the completed fixes. @cv @ericksoa, the environment has no configured protection reviewers, so please use the typed resolve-control-plane fallback after reviewing this exact revision.

@prekshivyas

Copy link
Copy Markdown
Contributor

New exact head after the main-only merge: 290da9bc91270f2693a94d72c6c165f48bef1817 on base 4f64b0ac19d321c6a85c88799c5675e33c3c1a7e. Standard CI is green, canonical GPT Advisor is 0/0/0, CodeRabbit is green, and no review threads remain. The Nemotron red is an Advisor protocol failure and is being rerun.

The current E2E gate now requires a maintainer-authorized run-control-plane dispatch for PR 6711 with this exact head/base and a review reason; the controller will run and verify the selected eight jobs. @cv @ericksoa, please authorize that exact revision. @cjagwani, the feature diff is unchanged by the main merge; please complete the requested re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery feature PR adds or expands user-visible functionality v0.0.83 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support multiple NemoClaw-managed instances on a single host

4 participants