[None][test] Use full-warp hidden_size (256) in allreduce CI smoke test#258
Open
MrGeva wants to merge 1 commit into
Open
[None][test] Use full-warp hidden_size (256) in allreduce CI smoke test#258MrGeva wants to merge 1 commit into
MrGeva wants to merge 1 commit into
Conversation
Diagnostic-only change: overrides the tiny test model's hidden_size from 64 (which launches an 8-thread, partial-warp block in the fused all-reduce RMSNorm kernel) to 256 (exactly one full warp), so the smoke test no longer exercises the partial-warp reduction path at all. Two prior CI runs of the base branch (agent/fix-ad-allreduce-ci) failed with an identical CUBLAS_STATUS_EXECUTION_FAILED error in an unrelated cuBLAS GEMM (torch_linear_simple), on the same node (nsc-svg-slurm-1-gpu-120), at a different allreduce strategy each time. This branch isolates whether that failure is specific to the partial-warp kernel path or is unrelated node/infra flakiness that also reproduces on the full-warp path. Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>
Author
|
/bot run --stage-list "DGX_B200-4_GPUs-AutoDeploy-1" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stacked on agent/fix-ad-allreduce-ci (upstream NVIDIA#16111). The only change: overrides the AutoDeploy allreduce-strategy smoke test's tiny model
hidden_sizefrom 64 to 256, so the fused all-reduce/RMSNorm kernel'sblockReduceSumalways launches exactly one full warp (32 threads) instead of the partial-warp (8-thread) block.Motivation
Diagnostic follow-up to NVIDIA#16111's CI investigation. Two
DGX_B200-4_GPUs-AutoDeploy-1runs of NVIDIA#16111 failed withCUBLAS_STATUS_EXECUTION_FAILEDin an unrelated cuBLAS GEMM (torch_linear_simple), on the same node (nsc-svg-slurm-1-gpu-120), at a different allreduce strategy each time - evidence pointing at node-level flakiness rather than the kernel fix itself. The same diff, opened separately as NVIDIA#16333 (againstmain), passed cleanly on a different node (nsc-svg-slurm-1-gpu-135), all 6 strategies.This PR isolates the single-file test diff on its own, stacked directly on the fix branch, for a clean before/after comparison.
Test Coverage
Same
test_allreduce_strategiessmoke test, all 6 strategies,hidden_size=256instead of 64.🤖 Generated with Claude Code