Repository navigation
[PyTorch] Run NVFP4 FC2 fprop and FC1 dgrad of the fused grouped MLP on CuTe DSL - #3871
Merged
timmoon10 merged 5 commits intoNov 21, 2026
Merged
Conversation
Contributor
[Medium impact] The changes since the previous review appear safe to merge. SummaryThe PR moves NVFP4 FC2 forward and FC1 input-gradient GEMMs to CuTe DSL when supported.
Reviews (6) · Last reviewed commit: "[PyTorch] Parametrize the grouped MLP CU..." · Reviewed by Greptile |
…on CuTe DSL The fused grouped MLP runs the NVFP4 FC2 forward and FC1 input-gradient GEMMs on cuBLAS. Run them on the cuDNN frontend CuTe DSL grouped GEMM kernel instead, like the MXFP8 path. Add NVTE_DISABLE_CUTEDSL_PLAIN_GROUPED_MLP_GEMM to move the GEMMs without a fused activation that both backends implement back to cuBLAS: the weight gradient GEMMs and the NVFP4 FC2 forward and FC1 input-gradient GEMMs. It replaces NVTE_DISABLE_CUTEDSL_WGRAD_FUSED_GROUPED_MLP, which covered the weight-gradient GEMMs only and stays as a deprecated alias, honored when the new variable is unset. A single-group MLP on the dense single-group path (every activation but the SReLU ones) keeps its dense GEMMs. NVFP4 with single_grouped_weight=True is not supported by the fused op (the cuBLAS path already rejects its unswizzled weight scales); the new paths raise NotImplementedError for it instead of failing in a kernel. Tests: backend routing for grouped and single-group MLPs, the environment variable and its alias, CUDA graph capture with NVFP4, and the single-grouped-weight error. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
xrennvidia
force-pushed
the
xren/nvfp4_cutedsl_fc2_fprop_fc1_dgrad
branch
from
November 17, 2026 13:09
81aab0c to
f20e8ad
Compare
timmoon10
requested changes
Nov 20, 2026
… dgrad Pass the operands of the NVFP4 FC2 fprop and FC1 dgrad CuTe DSL GEMMs in cuDNN's canonical layouts instead of reshaping them into the kernel-facing strided views, and drop the reshape helper from cpp_extensions/gemm.py. These two GEMMs fall back to cuBLAS when the cuDNN frontend predates canonical layouts, whatever the plain-GEMM backend setting. Reject NVFP4 with single_grouped_weight=True once, up front, so the error does not depend on the GEMM backend. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Rename NVTE_DISABLE_CUTEDSL_PLAIN_GROUPED_MLP_GEMM to NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP, since "plain GEMM" does not say which GEMMs are meant, and rename the helpers and tests to match. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Run the MXFP8 FC2 fprop and FC1 dgrad GEMMs of the fused grouped MLP on cuBLAS when NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP is set, as is already done for NVFP4 and for the wgrad GEMMs. A dense single group now takes the branch shared with cuBLAS for both recipes, so drop the unreachable dense case from the MXFP8 kernel branch. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Turn test_grouped_mlp_cuda_graph_safe_mxfp8 into test_grouped_mlp_cuda_graph_safe, parametrized over MXFP8 and NVFP4, and drop the NVFP4 wrapper that called the MXFP8 test. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Member
|
/te-ci pytorch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The fused grouped MLP runs the NVFP4 FC2 forward and FC1 input-gradient GEMMs on cuBLAS,
while every other GEMM in the op already uses a cuDNN frontend CuTe DSL kernel. This PR
runs those two on the CuTe DSL grouped GEMM kernel as well, like the MXFP8 path, so that
with NVFP4 all grouped GEMMs of the op use CuTe DSL by default.
It also adds one environment variable that moves the unfused GEMMs of the op (FC2 fprop,
FC1 dgrad and wgrad) back to cuBLAS, for both NVFP4 and MXFP8.
Fixes # (issue)
Type of change
Changes
grouped_gemm_quant_wrapper_sm100.The operands are passed in cuDNN's canonical layouts ([PyTorch] Pass canonical layouts to cuDNN grouped GEMM in fused grouped MLP #3641, [PyTorch] Pass canonical layouts to cuDNN for the NVFP4 fused grouped MLP #3820), with no reshapes.
The GEMM alpha uses
_nvfp4_global_scale_pair([PyTorch] Fuse NVFP4 global scales and GEMM alpha in the grouped MLP #3756).supports_canonical_layouts(1.32.1 or later). On an older frontend they run on cuBLAS.NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP(default0). Set to1, it movesthe GEMMs without a fused activation back to cuBLAS: the FC2 fprop, FC1 dgrad and
weight-gradient GEMMs, for both NVFP4 and MXFP8.
NVTE_DISABLE_CUTEDSL_WGRAD_FUSED_GROUPED_MLP, which covered the weight-gradient GEMMsonly, stays as a deprecated alias: honored when the new variable is unset, with a
DeprecationWarning.ones) keeps its dense GEMMs either way.
single_grouped_weight=Trueis not supported by the fused op. It now raisesNotImplementedErrorat the start of the forward pass, whichever backend is selected,instead of failing inside a kernel.
docs/envvars.rst.Testing
On GB300 (SM 10.3), cuDNN frontend 1.34.0:
tests/pytorch/test_grouped_mlp.py: 1394 passed, 3948 skipped, both with the variableunset and with
NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP=1.tests/pytorch/nvfp4/test_nvfp4_global_scale.py: 112 passed.supports_canonical_layoutsfrom the quant wrapper, not on a real older frontend.New and changed tests:
test_grouped_mlp_unfused_gemm_backend: which backend each of the four GEMMs uses, withthe variable unset and set, for grouped and single-group MLPs and three activations, for
NVFP4 and for MXFP8 with discrete and single grouped weights.
test_cudnn_unfused_gemm_disabled_env_var: the variable, its deprecated alias andprecedence.
test_grouped_mlp_cuda_graph_safe: the formertest_grouped_mlp_cuda_graph_safe_mxfp8,now parametrized over MXFP8 and NVFP4.
test_grouped_mlp_nvfp4_rejects_single_grouped_weight.The existing
test_grouped_mlpNVFP4 matrix now runs through the new paths by default.Checklist: