Skip to content

fix(cli): fall back to a writable state dir when $HOME is unwritable#805

Open
albcui wants to merge 1 commit into
mainfrom
albcui/harden-nemo-services-state-dir-fallback
Open

fix(cli): fall back to a writable state dir when $HOME is unwritable#805
albcui wants to merge 1 commit into
mainfrom
albcui/harden-nemo-services-state-dir-fallback

Conversation

@albcui

@albcui albcui commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

nemo services run writes per-instance lock/descriptor state under $XDG_STATE_HOME (default ~/.local/state/nmp). When the nmp-api entrypoint runs under a uid that does not own $HOME -- a common Kubernetes/OpenShift hardening pattern (arbitrary runAsUser) -- that location is unwritable and the command crashes with PermissionError before starting.

Make _base_state_dir() resilient: probe writability non-destructively and fall back to a per-uid dir under the system temp dir when the XDG/$HOME location cannot be written. The probe tolerates an unsearchable parent (a $HOME created mode 0700 for a different uid raises EACCES on stat) so it never propagates the very error the fallback exists to avoid.

Summary by CodeRabbit

  • Bug Fixes
    • Improved service state-directory handling when the preferred location is unavailable or unwritable.
    • Added a secure temporary fallback location to help services continue operating in restricted environments.
    • Ensured state directories remain consistent throughout a process and instance operations avoid failures caused by unwritable home directories.

@github-actions github-actions Bot added the fix label Jul 21, 2026
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 26413/34063 77.5% 61.8%
Integration Tests 15181/32688 46.4% 18.7%

`nemo services run` writes per-instance lock/descriptor state under
$XDG_STATE_HOME (default ~/.local/state/nmp). When the nmp-api entrypoint
runs under a uid that does not own $HOME -- a common Kubernetes/OpenShift
hardening pattern (arbitrary runAsUser) -- that location is unwritable and
the command crashes with PermissionError before starting.

Resolve the base dir by actually creating its `instances/` dir, falling back
to a per-uid dir under the system temp dir when the preferred XDG/$HOME
location can't be written. Selecting by a real mkdir (rather than an advisory
os.access probe) is correct even where the two disagree -- NFS root-squash,
SELinux -- and even if the fallback itself is unwritable. The result is
memoized so reads and writes agree on one directory, and the temp fallback is
created owner-only (0700) so the descriptor (pid/port/config path) and service
log are not world-readable on a shared /tmp.

Regenerate the vendored SDK CLI copy. Add regression tests for fallback
resolution, memoization, the owner-only temp dir, the no-usable-tempdir case,
and the no-writable-location error.

Signed-off-by: Albert Cui <albcui@nvidia.com>
@albcui
albcui force-pushed the albcui/harden-nemo-services-state-dir-fallback branch from c3cf273 to d87108d Compare July 21, 2026 00:23
@albcui
albcui marked this pull request as ready for review July 21, 2026 00:23
@albcui
albcui requested review from a team as code owners July 21, 2026 00:23
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

State directory resolution

Layer / File(s) Summary
Resolution and fallback behavior
packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/services/_process.py
State resolution now probes writable XDG locations, falls back to a per-UID temporary directory, restricts fallback permissions, logs fallback use, caches the result, and raises when no location is available.
Resolution regression coverage
packages/nemo_platform_ext/tests/cli/commands/test_services_process.py
Tests cover preferred-path creation, memoization, fallback behavior and permissions, unavailable temporary storage, failure reporting, and unwritable home directories.

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant _base_state_dir
  participant Filesystem
  participant Logger
  Caller->>_base_state_dir: request base state directory
  _base_state_dir->>Filesystem: probe preferred instances directory
  Filesystem-->>_base_state_dir: writable or unavailable
  _base_state_dir->>Filesystem: probe per-UID fallback directory
  Filesystem-->>_base_state_dir: writable fallback
  _base_state_dir->>Logger: emit fallback warning
  _base_state_dir-->>Caller: return memoized directory
Loading

Suggested reviewers: mckornfield

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main change: falling back to a writable state directory when the HOME-based location is unwritable.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch albcui/harden-nemo-services-state-dir-fallback

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/services/_process.py`:
- Around line 141-143: Update the warning in _preferred_base_state_dir()’s
state-directory fallback path to remove the unsupported _NMP_STATE_DIR override
from the message, leaving only configuration variables that actually affect
resolution.
- Around line 104-105: Remove the contextlib.suppress around os.chmod for the
fallback nmp-state directory so EPERM or other chmod failures reject the
candidate and prevent using an unsecured path. Update the fallback handling
around this chmod operation to continue safely without writing descriptors or
logs there, and add a regression test covering a pre-created squatter-controlled
directory whose chmod fails.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d9a315a0-f5e6-4f51-b7eb-a00e0eef4ebd

📥 Commits

Reviewing files that changed from the base of the PR and between 2c1a9ef and d87108d.

⛔ Files ignored due to path filters (2)
  • sdk/python/nemo-platform/src/nemo_platform/cli/commands/services/_process.py is excluded by !sdk/**
  • sdk/python/nemo-platform/tests/vendored/nemo_platform_ext/cli/commands/test_services_process.py is excluded by !sdk/**
📒 Files selected for processing (2)
  • packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/services/_process.py
  • packages/nemo_platform_ext/tests/cli/commands/test_services_process.py

Comment on lines +104 to +105
with contextlib.suppress(OSError):
os.chmod(base.parent, 0o700) # the `nmp-state-<uid>` dir under the temp root

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Reject fallback directories that cannot be secured.

A different user can pre-create /tmp/nmp-state-<uid> as writable. chmod then fails with EPERM, is suppressed, and descriptors/logs may be written under the squatter-controlled path. Let the chmod failure reject this fallback candidate and add a squatting regression test.

Also applies to: 139-147

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/services/_process.py`
around lines 104 - 105, Remove the contextlib.suppress around os.chmod for the
fallback nmp-state directory so EPERM or other chmod failures reject the
candidate and prevent using an unsecured path. Update the fallback handling
around this chmod operation to continue safely without writing descriptors or
logs there, and add a regression test covering a pre-created squatter-controlled
directory whose chmod fails.

Comment on lines +141 to +143
logger.warning(
"State directory %s is not writable; using %s instead. Set XDG_STATE_HOME "
"or _NMP_STATE_DIR to a writable path to control where instance state lives.",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Do not advertise an ignored override.

_preferred_base_state_dir() reads only XDG_STATE_HOME and $HOME; _NMP_STATE_DIR will not change resolution. Remove it from this warning or implement it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/services/_process.py`
around lines 141 - 143, Update the warning in _preferred_base_state_dir()’s
state-directory fallback path to remove the unsupported _NMP_STATE_DIR override
from the message, leaving only configuration variables that actually affect
resolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant