Skip to content

[PyTorch] Run NVFP4 FC2 fprop and FC1 dgrad of the fused grouped MLP on CuTe DSL - #3871

Merged
timmoon10 merged 5 commits into
NVIDIA:mainfrom
xrennvidia:xren/nvfp4_cutedsl_fc2_fprop_fc1_dgrad
Nov 21, 2026
Merged

timmoon10 merged 5 commits into
NVIDIA:mainfrom
xrennvidia:xren/nvfp4_cutedsl_fc2_fprop_fc1_dgrad

Conversation

@xrennvidia

@xrennvidia xrennvidia commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The fused grouped MLP runs the NVFP4 FC2 forward and FC1 input-gradient GEMMs on cuBLAS,
while every other GEMM in the op already uses a cuDNN frontend CuTe DSL kernel. This PR
runs those two on the CuTe DSL grouped GEMM kernel as well, like the MXFP8 path, so that
with NVFP4 all grouped GEMMs of the op use CuTe DSL by default.

It also adds one environment variable that moves the unfused GEMMs of the op (FC2 fprop,
FC1 dgrad and wgrad) back to cuBLAS, for both NVFP4 and MXFP8.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Run the NVFP4 FC2 fprop and FC1 dgrad GEMMs through grouped_gemm_quant_wrapper_sm100.
    The operands are passed in cuDNN's canonical layouts ([PyTorch] Pass canonical layouts to cuDNN grouped GEMM in fused grouped MLP #3641, [PyTorch] Pass canonical layouts to cuDNN for the NVFP4 fused grouped MLP #3820), with no reshapes.
    The GEMM alpha uses _nvfp4_global_scale_pair ([PyTorch] Fuse NVFP4 global scales and GEMM alpha in the grouped MLP #3756).
  • These two NVFP4 GEMMs need a cuDNN frontend whose quant wrapper advertises
    supports_canonical_layouts (1.32.1 or later). On an older frontend they run on cuBLAS.
  • Add NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP (default 0). Set to 1, it moves
    the GEMMs without a fused activation back to cuBLAS: the FC2 fprop, FC1 dgrad and
    weight-gradient GEMMs, for both NVFP4 and MXFP8.
  • NVTE_DISABLE_CUTEDSL_WGRAD_FUSED_GROUPED_MLP, which covered the weight-gradient GEMMs
    only, stays as a deprecated alias: honored when the new variable is unset, with a
    DeprecationWarning.
  • A single-group MLP on the dense single-group path (every activation except the SReLU
    ones) keeps its dense GEMMs either way.
  • NVFP4 with single_grouped_weight=True is not supported by the fused op. It now raises
    NotImplementedError at the start of the forward pass, whichever backend is selected,
    instead of failing inside a kernel.
  • Document the environment variable in docs/envvars.rst.

Testing

On GB300 (SM 10.3), cuDNN frontend 1.34.0:

  • tests/pytorch/test_grouped_mlp.py: 1394 passed, 3948 skipped, both with the variable
    unset and with NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP=1.
  • tests/pytorch/nvfp4/test_nvfp4_global_scale.py: 112 passed.
  • The cuBLAS fallback for a frontend without canonical layouts was exercised by removing
    supports_canonical_layouts from the quant wrapper, not on a real older frontend.

New and changed tests:

  • test_grouped_mlp_unfused_gemm_backend: which backend each of the four GEMMs uses, with
    the variable unset and set, for grouped and single-group MLPs and three activations, for
    NVFP4 and for MXFP8 with discrete and single grouped weights.
  • test_cudnn_unfused_gemm_disabled_env_var: the variable, its deprecated alias and
    precedence.
  • test_grouped_mlp_cuda_graph_safe: the former test_grouped_mlp_cuda_graph_safe_mxfp8,
    now parametrized over MXFP8 and NVFP4.
  • test_grouped_mlp_nvfp4_rejects_single_grouped_weight.

The existing test_grouped_mlp NVFP4 matrix now runs through the new paths by default.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 15, 2026
@greptile-apps

greptile-apps Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The changes since the previous review appear safe to merge.

Summary

The PR moves NVFP4 FC2 forward and FC1 input-gradient GEMMs to CuTe DSL when supported.

  • Grouped NVFP4 MLPs can run FC2 forward and FC1 input gradients on CuTe DSL.
  • One setting routes the unfused grouped MLP GEMMs to cuBLAS.
  • NVFP4 fused MLPs reject single grouped weights with a clear exception.

Reviews (6) · Last reviewed commit: "[PyTorch] Parametrize the grouped MLP CU..." · Reviewed by Greptile

…on CuTe DSL

The fused grouped MLP runs the NVFP4 FC2 forward and FC1 input-gradient
GEMMs on cuBLAS. Run them on the cuDNN frontend CuTe DSL grouped GEMM
kernel instead, like the MXFP8 path.

Add NVTE_DISABLE_CUTEDSL_PLAIN_GROUPED_MLP_GEMM to move the GEMMs without a
fused activation that both backends implement back to cuBLAS: the weight
gradient GEMMs and the NVFP4 FC2 forward and FC1 input-gradient GEMMs. It
replaces NVTE_DISABLE_CUTEDSL_WGRAD_FUSED_GROUPED_MLP, which covered the
weight-gradient GEMMs only and stays as a deprecated alias, honored when
the new variable is unset. A single-group MLP on the dense single-group
path (every activation but the SReLU ones) keeps its dense GEMMs.

NVFP4 with single_grouped_weight=True is not supported by the fused op
(the cuBLAS path already rejects its unswizzled weight scales); the new
paths raise NotImplementedError for it instead of failing in a kernel.

Tests: backend routing for grouped and single-group MLPs, the environment
variable and its alias, CUDA graph capture with NVFP4, and the
single-grouped-weight error.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
@xrennvidia
xrennvidia force-pushed the xren/nvfp4_cutedsl_fc2_fprop_fc1_dgrad branch from 81aab0c to f20e8ad Compare November 17, 2026 13:09
Comment thread docs/envvars.rst Resolved
Comment thread tests/pytorch/test_grouped_mlp.py Resolved
Comment thread tests/pytorch/test_grouped_mlp.py Resolved
xrennvidia and others added 4 commits November 20, 2026 00:24
… dgrad

Pass the operands of the NVFP4 FC2 fprop and FC1 dgrad CuTe DSL GEMMs in
cuDNN's canonical layouts instead of reshaping them into the kernel-facing
strided views, and drop the reshape helper from cpp_extensions/gemm.py.

These two GEMMs fall back to cuBLAS when the cuDNN frontend predates
canonical layouts, whatever the plain-GEMM backend setting.

Reject NVFP4 with single_grouped_weight=True once, up front, so the error
does not depend on the GEMM backend.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Rename NVTE_DISABLE_CUTEDSL_PLAIN_GROUPED_MLP_GEMM to
NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP, since "plain GEMM" does not
say which GEMMs are meant, and rename the helpers and tests to match.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Run the MXFP8 FC2 fprop and FC1 dgrad GEMMs of the fused grouped MLP on
cuBLAS when NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP is set, as is
already done for NVFP4 and for the wgrad GEMMs.

A dense single group now takes the branch shared with cuBLAS for both
recipes, so drop the unreachable dense case from the MXFP8 kernel branch.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Turn test_grouped_mlp_cuda_graph_safe_mxfp8 into
test_grouped_mlp_cuda_graph_safe, parametrized over MXFP8 and NVFP4, and
drop the NVFP4 wrapper that called the MXFP8 test.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
@timmoon10

Copy link
Copy Markdown
Member

/te-ci pytorch

@timmoon10 timmoon10 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending CI

@timmoon10
timmoon10 merged commit e8356c0 into NVIDIA:main Nov 21, 2026
22 of 23 checks passed
@xrennvidia
xrennvidia deleted the xren/nvfp4_cutedsl_fc2_fprop_fc1_dgrad branch November 22, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants