fix(models): point LoRA sidecar to write nmp state to writable scratch volume#798
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
albcui
force-pushed
the
albcui/nvbugs/fix-lora-sidecar-unwritable-state-dir
branch
from
July 20, 2026 22:43
4ab8e1e to
1c6932d
Compare
Contributor
|
albcui
force-pushed
the
albcui/nvbugs/fix-lora-sidecar-unwritable-state-dir
branch
from
July 20, 2026 22:44
1c6932d to
a8b5a72
Compare
albcui
force-pushed
the
albcui/nvbugs/fix-lora-sidecar-unwritable-state-dir
branch
from
July 20, 2026 23:11
642f755 to
8f2ab22
Compare
albcui
marked this pull request as ready for review
July 20, 2026 23:16
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe deployments-plugin compiler configures the LoRA adapters sidecar to store service state under ChangesWritable service state paths
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/services/_process.py`:
- Around line 135-141: Update the warning in the state-directory resolver to
remove the unsupported _NMP_STATE_DIR reference and advise users to set only
XDG_STATE_HOME, while preserving the existing preferred and fallback values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8fa2c9ee-d272-4b54-92f1-69ca35058534
⛔ Files ignored due to path filters (2)
sdk/python/nemo-platform/src/nemo_platform/cli/commands/services/_process.pyis excluded by!sdk/**sdk/python/nemo-platform/tests/vendored/nemo_platform_ext/cli/commands/test_services_process.pyis excluded by!sdk/**
📒 Files selected for processing (4)
packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/services/_process.pypackages/nemo_platform_ext/tests/cli/commands/test_services_process.pyservices/core/models/src/nmp/core/models/controllers/backends/k8s_nim_operator/reconcilers/k8s.pyservices/core/models/tests/unit/controllers/test_k8s_nim_operator_backend.py
albcui
force-pushed
the
albcui/nvbugs/fix-lora-sidecar-unwritable-state-dir
branch
3 times, most recently
from
July 21, 2026 00:09
5a2aaf7 to
f798f15
Compare
albcui
force-pushed
the
albcui/nvbugs/fix-lora-sidecar-unwritable-state-dir
branch
from
July 21, 2026 18:46
f798f15 to
fbb164b
Compare
The adapters sidecar runs the nmp-api image via `nemo services run --sidecars adapters`, but the pod's securityContext runs it as the vLLM uid (2000), which does not own that image's baked $HOME (/home/nvs, uid 1000). `nemo services run` writes per-instance lock/descriptor state under $XDG_STATE_HOME (default ~/.local/state), so the sidecar hit PermissionError creating /home/nvs/.local and crash-looped to ERROR, blocking LoRA adapter inference. Point the sidecar's XDG_STATE_HOME at the writable scratch volume (/scratch/.nmp-state), outside the /scratch/loras subtree the adapters controller GCs. `nemo services run` already honors XDG_STATE_HOME, so no CLI change is needed. Re-homed onto the deployments_plugin cutover (#705), which removed the k8s_nim_operator backend the original fix patched; the redirect now lives in deployments_plugin/compiler.py::_lora_sidecar and is covered by that backend's compiler unit test. The XDG_STATE_HOME redirect itself was validated end-to-end on minikube earlier (a uid-2000 sidecar crash-loops without it and starts cleanly with it); this commit emits that same env from the new backend. Signed-off-by: Albert Cui <albcui@nvidia.com>
albcui
force-pushed
the
albcui/nvbugs/fix-lora-sidecar-unwritable-state-dir
branch
from
July 21, 2026 18:57
fbb164b to
98b09e2
Compare
albcui
changed the title
anubhutivyas
approved these changes
Jul 21, 2026
albcui
enabled auto-merge
July 21, 2026 19:08
soluwalana
pushed a commit
that referenced
this pull request
Jul 22, 2026
…#842) * fix(models): redirect LoRA sidecar XDG_DATA_HOME to a writable volume Follow-up to #798, which redirected only $XDG_STATE_HOME. The adapters sidecar's `nemo services run` also resolves its local data dir (SQLite DB, encryption key) via nmp_user_data_dir(), which honors $XDG_DATA_HOME (default ~/.local/share/nemo) -- a distinct XDG home from $XDG_STATE_HOME (default ~/.local/state). The pod runs the sidecar as the vLLM uid (2000), which does not own the nmp-api image's baked $HOME (/home/nvs, uid 1000), so startup still crash-looped with `PermissionError: [Errno 13] Permission denied: '/home/nvs/.local'` when mkdir'ing ~/.local/share/nemo. Point both XDG homes at a single writable scratch base (/scratch/.local), named for the ~/.local these homes normally live under. State lands under .../nmp/ and data under .../nemo/, so one base holds both without collision, and it stays outside the /scratch/loras subtree the adapters controller GCs. Covered by the deployments_plugin compiler unit test. Signed-off-by: Albert Cui <albcui@nvidia.com> * Make comments less verbose Signed-off-by: Albert Cui <albcui@nvidia.com> --------- Signed-off-by: Albert Cui <albcui@nvidia.com> Signed-off-by: Sam Oluwalana <soluwalana@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LoRA-enabled deployments never errors with
PermissionErrorbecause the adapter sidecar runs asuid 2000(controlled by the pod's securityContext), but thenmp-apiimage's baked-inHOME=/home/nvsis owned byuid 1000. The sidecar tries to runnemo services run --sidecars adapters, which tries to write its state to$XDG_STATE_HOME(default/home/nvs/.local/state), whichuid 2000doesn't have access to.The fix is to point
$XDG_STATE_HOMEto a directory that the sidecar does have access to, which is the locally mounted/scratch.Summary by CodeRabbit
Bug Fixes
Tests