[TRTLLMINF-132][infra] Pre-bake fat sqsh to eliminate GPU idle time during CI pip installs#15398
[TRTLLMINF-132][infra] Pre-bake fat sqsh to eliminate GPU idle time during CI pip installs#15398EmmaQiaoCh wants to merge 17 commits into
Conversation
…pip installs
Add a "fat sqsh" caching layer for SLURM/ENROOT CI jobs. The fat sqsh is
an enroot .sqsh image pre-baked with TensorRT-LLM installed (base image +
pip install requirements-dev.txt + wheel). GPU test jobs that reuse a cached
fat sqsh skip the 10-18 min pip install phase entirely.
Key design:
- Cache key: sha256(llmTarfile URL)[:16] — encodes commit + platform
- Fat sqsh dir: ${cluster.scratchPath}/users/svc_tensorrt/fat_sqsh/
- First job: imports base sqsh, builds fat sqsh inline in srunPrologue via
fat_build_inline.sh; subsequent jobs reuse directly (SKIP_INSTALL=1)
- Best-effort flock prevents duplicate concurrent builds; failures fall back
to base sqsh + full slurm_install.sh (non-fatal)
- Atomic publish via mv -f from .tmp; age-prune after 7 days
- SKIP_INSTALL env var passed into container via --container-env; checked in
slurm_install.sh to skip pip installs while still downloading the tarball
(needed for source files by pytest)
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
/bot run |
📝 WalkthroughWalkthroughAdds a shared "fat sqsh" caching mechanism to the Jenkins ENROOT SLURM pipeline. A new ChangesFat sqsh caching for ENROOT SLURM jobs
Sequence DiagramsequenceDiagram
participant Jenkins as Jenkins Pipeline
participant Remote as Remote SLURM Node
participant Enroot as Enroot Runtime
participant FatSqsh as Shared fat-hash.sqsh
Jenkins->>Remote: upload fat_build_inline.sh
Jenkins->>Remote: emit ENROOT launcher prologue
Remote->>FatSqsh: check fat-<hash>.sqsh exists?
alt fat sqsh exists
FatSqsh-->>Remote: reuse fat-<hash>.sqsh, SKIP_INSTALL=1
else fat sqsh missing
Remote->>Enroot: import base sqsh (per-job)
Remote->>Remote: flock: run fat_build_inline.sh
alt fat build succeeds
Enroot->>FatSqsh: atomic rename tmp sqsh → fat-<hash>.sqsh
Remote->>Remote: switch to fat sqsh, SKIP_INSTALL=1
else fat build fails
Remote->>Remote: fall back to per-job base sqsh
end
end
Remote->>Enroot: srun --container-env=SKIP_INSTALL
Enroot->>Remote: slurm_install.sh checks SKIP_INSTALL, skips pip if set
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/L0_Test.groovy`:
- Around line 1266-1271: The fatHash calculation for the fat-sqsh cache key only
includes llmTarfile, but the fat sqsh is built from both the base container
image and the llmTarfile. Modify the fatHash computation to include the base
image (LLM_DOCKER_IMAGE) along with llmTarfile in the hash calculation to ensure
the cache key reflects all build-affecting inputs. This prevents reusing an
incompatible cached sqsh when the same tar URL is used with a different base
image. Additionally, consider including Ray or install profile information in
the key if fat_build_inline.sh does not unconditionally bake those dependencies.
In `@jenkins/scripts/fat_build_inline.sh`:
- Around line 1-11: The file fat_build_inline.sh is missing the required NVIDIA
copyright header block. Add the appropriate NVIDIA copyright and license header
to the beginning of the file, immediately after the shebang (#!/bin/bash line).
The header should include copyright notice and reference to the applicable open
source license for TensorRT-LLM, consistent with other source files in the
repository.
- Around line 32-33: The echo statement at line 32-33 logs the full
LLM_TARFILE_URL variable, which may contain sensitive pre-signed tokens or
credentials in query parameters. Instead of logging the complete URL directly,
sanitize or redact the URL before printing it to the log. Extract only the base
URL portion (without query parameters) or use a redaction method to mask the
sensitive parts while keeping enough information for debugging purposes. Replace
the direct reference to $LLM_TARFILE_URL in the echo statement with a sanitized
version that removes credential-bearing query parameters.
In `@jenkins/scripts/slurm_install.sh`:
- Line 31: The variable `pytestCommand` at line 31 in the condition `if [[
$pytestCommand == *--run-ray* ]]; then` is accessed without protection against
undefined variable errors, which violates the `set -u` (nounset) mode enabled by
`set -xEeuo pipefail`. Since `pytestCommand` may not be assigned until later
(line 63), the check at line 31 will fail when running in non-Disaggregated mode
where `slurm_install_setup()` is called before the assignment. Replace the
unguarded variable reference with a safe parameter expansion that provides a
default value, such as `${pytestCommand:-}` or `${pytestCommand:+...}`, to
prevent triggering a nounset error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 20869e82-454c-4829-b17b-6574e45ce622
📒 Files selected for processing (3)
jenkins/L0_Test.groovyjenkins/scripts/fat_build_inline.shjenkins/scripts/slurm_install.sh
|
PR_Github #54495 [ run ] triggered by Bot. Commit: |
- Include LLM_DOCKER_IMAGE in fat sqsh hash key so a base image change
with the same tarfile URL correctly busts the cache
- Add NVIDIA SPDX copyright header to fat_build_inline.sh
- Sanitize tarfile URL before logging (strip query params) to avoid
leaking any credential-bearing query parameters
- Use ${pytestCommand:-} to guard against nounset failure under set -u
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
PR_Github #54495 [ run ] completed with state
|
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
/bot run |
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
/bot run |
|
PR_Github #58018 [ run ] triggered by Bot. Commit: |
|
PR_Github #58018 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #58114 [ run ] triggered by Bot. Commit: |
|
PR_Github #58114 [ run ] completed with state
|
|
/bot run --stage-list "B300-PyTorch-2, DGX_A100-FMHA-Post-Merge-1, DGX_B200-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #58386 [ run ] triggered by Bot. Commit: |
|
PR_Github #58386 [ run ] completed with state
|
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
|
/bot run --stage-list "B300-PyTorch-2, DGX_A100-FMHA-Post-Merge-1, DGX_B200-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #59141 [ run ] triggered by Bot. Commit: |
|
PR_Github #59141 [ run ] completed with state
|
|
/bot run --stage-list "DGX_B200-AutoDeploy-1" |
1 similar comment
|
/bot run --stage-list "DGX_B200-AutoDeploy-1" |
|
PR_Github #59578 [ run ] triggered by Bot. Commit: |
|
PR_Github #59578 [ run ] completed with state
|
Fat sqsh was installing TRT-LLM into /opt/trtllm/ inside the container, but GPU test jobs expect the source tree at /tmp/TensorRT-LLM/src (LLM_ROOT=/tmp/TensorRT-LLM/src). Change fat_build_inline.sh to cd /tmp before extracting the tarball so the layout matches the non-fat-sqsh path. Update slurm_install.sh SKIP_INSTALL branch accordingly (resourcePathNode=/tmp). Also add tail -f of the CPU builder's SLURM log during the Prepare Container poll loop so package versions and install progress are visible in Jenkins output. Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
/bot run --stage-list "DGX_A100-FMHA-Post-Merge-1, DGX_B200-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #59605 [ run ] triggered by Bot. Commit: |
|
/bot run --stage-list "DGX_B200-AutoDeploy-1" |
|
PR_Github #59625 [ run ] triggered by Bot. Commit: |
|
PR_Github #59605 [ run ] completed with state |
|
PR_Github #59625 [ run ] completed with state
|
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
/bot run --stage-list "DGX_B200-AutoDeploy-1" |
|
PR_Github #59654 [ run ] triggered by Bot. Commit: |
|
PR_Github #59654 [ run ] completed with state
|
|
/bot run --stage-list "DGX_B200-AutoDeploy-1" |
|
PR_Github #59682 [ run ] triggered by Bot. Commit: |
|
PR_Github #59682 [ run ] completed with state |
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
/bot run --stage-list "DGX_B200-AutoDeploy-1" |
|
PR_Github #59702 [ run ] triggered by Bot. Commit: |
|
PR_Github #59702 [ run ] completed with state |
|
/bot run --stage-list "DGX_B200" --disable-fail-fast |
|
PR_Github #59837 [ run ] triggered by Bot. Commit: |
|
PR_Github #59837 [ run ] completed with state
|
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
|
/bot run --stage-list "DGX_B200-AutoDeploy-1" |
|
PR_Github #59945 [ run ] triggered by Bot. Commit: |
|
PR_Github #59945 [ run ] completed with state
|
2 participants
…pip installs
Add a "fat sqsh" caching layer for SLURM/ENROOT CI jobs. The fat sqsh is an enroot .sqsh image pre-baked with TensorRT-LLM installed (base image + pip install requirements-dev.txt + wheel). GPU test jobs that reuse a cached fat sqsh skip the 10-18 min pip install phase entirely.
Key design:
Summary by CodeRabbit
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.