Skip to content

[PyTorch] Clone quantized tensor scaling buffers independently - #3874

Open
ZenAlexa wants to merge 4 commits into
NVIDIA:mainfrom
ZenAlexa:contrib/clone-quantization-metadata
Open

ZenAlexa wants to merge 4 commits into
NVIDIA:mainfrom
ZenAlexa:contrib/clone-quantization-metadata

Conversation

@ZenAlexa

@ZenAlexa ZenAlexa commented Nov 16, 2026 •

Copy link
Copy Markdown

Description

clone() copies the packed payload of the quantized tensor classes and inherits their scaling buffers by reference. A later copy_() into the clone can therefore change the original tensor's decoded values. With Float8CurrentScalingQuantizer, cloning a tensor of ones and copying a separately quantized tensor of twos into the clone changes the original tensor from 1 to 2 on current main.

Type of change

  • Bug fix

Changes

  • Clone the tensor-backed storage fields for Float8, MXFP8, NVFP4, and blockwise Float8, including their scaling buffers and NVFP4 amax tensors.
  • Preserve Float8 transpose-cache validity and support NVFP4 columnwise-only storage in clone().
  • Route torch.clone() through the NVFP4 and blockwise Float8 clone methods so Hybrid tensor cloning reaches the same storage-copy contract.
  • Keep get_metadata(), _IdentityFunc, detach(), and view() unchanged. Quantizer construction retains its existing configuration-copy and shared calibration-state behavior.

Reproduction

import torch
import transformer_engine.pytorch as te

q = te.Float8CurrentScalingQuantizer(
    fp8_dtype=te.DType.kFloat8E4M3,
    device=torch.device("cuda", torch.cuda.current_device()),
    rowwise=True,
    columnwise=True,
)
original = q(torch.ones((128, 128), device="cuda"))
before = original.dequantize().clone()
copied = original.clone()
copied.copy_(q(torch.full((128, 128), 2.0, device="cuda")))
print(before[0, 0].item())
print(original.dequantize()[0, 0].item())
print(copied.dequantize()[0, 0].item())

Current main prints 1.0, 2.0, 2.0. The patch prints 1.0, 1.0, 2.0.

Validation

After merging main 925832d7 on November 17, all 42 CUDA clone lifecycles passed again. The quantized-tensor implementation and scenario are unchanged, and both the clone scenario and main's new swizzle test remain in the L0 launcher. The official TE 2.21 extension and device limits below still apply.

  • Executed the quantizer, clone, copy, and dequantize sequence above with real CUDA tensors and the native TE extension.
  • Exercised CPU and CUDA-buffer lifecycles for all four tensor classes, including rowwise, columnwise, and both layouts, scalar and row-scaled NVFP4 amax, buffer independence, unchanged original values after copy_, detach/view aliases, quantizer calibration-state sharing, stale transpose state, and clone backward. The candidate passed 85 observations in each environment.
  • Added tests/pytorch/scenario_quantized_clone.py to the PyTorch L0 launcher. Its 42 CUDA lifecycles cover the method and torch.clone() entry points, both Hybrid storage directions, type preservation, independent buffers, clone replacement, unchanged originals, detach aliases, backward propagation, and native FP8 decoded values. The same committed scenario reports 42 failures on main and 42 passes with the patch.
  • Ran the existing test_identity_op and test_detach_preserves_subclass coverage. Both main and candidate reported 5 passed and 3 architecture skips.
  • Black 24.4.2 and pylint 3.3.1 passed for the changed files with the project settings.

The native run used an RTX 4090 Laptop GPU, SM 8.9, Python 3.12, Torch 2.8.0+cu129, and official TE 2.21.0 CUDA core and matching PyTorch-extension wheels with the current Python source. cuDNN was explicitly set to 9.12.0.46 to satisfy TE's import check, differing from the Torch wheel's 9.10.2.21 pin. This environment validates the Python storage behavior and native FP8 reproduction. It is separate from a complete current-main native build.

MXFP8, NVFP4, and blockwise FP8 CUDA-buffer checks cover storage operations on this device. Their architecture-specific quantize/GEMM paths, the full L0 suites, and Hopper/Blackwell coverage remain unrun. Please run the relevant L0 quantized-tensor and Hybrid coverage in the supported project environments.

Copy tensor-backed quantization metadata alongside packed payloads, keep
transpose-cache validity, and route clone dispatch to the owning
quantized tensor implementation.

Signed-off-by: Ziming Wang <zimingwang945@gmail.com>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 16, 2026
@ZenAlexa
ZenAlexa marked this pull request as ready for review November 16, 2026 13:56
@ZenAlexa
ZenAlexa requested a review from ksivaman as a code owner November 16, 2026 13:56
Copilot AI balanced review requested due to automatic review settings November 16, 2026 13:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@greptile-apps

greptile-apps Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues were found.

Summary

Quantized tensor clones now copy their scale buffers independently, so replacing a clone does not change the original tensor.

  • Quantized tensor clones keep their scaling buffers separate.

No new actionable issues found. CUDA tests were not run during this review.

Reviews (4) · Last reviewed commit: "[PyTorch] Merge current main" · Reviewed by Greptile

Run a standalone clone and copy scenario from the PyTorch L0 launcher.
Cover storage independence, clone updates, Hybrid children, detach aliases,
and gradient propagation across the quantized tensor formats.

Signed-off-by: Ziming Wang <zimingwang945@gmail.com>
Signed-off-by: Ziming Wang <zimingwang945@gmail.com>
Signed-off-by: Ziming Wang <zimingwang945@gmail.com>
@ZenAlexa

Copy link
Copy Markdown
Author

Rechecked the current head 1a7cb06b against the clone-independence finding. tests/pytorch/scenario_quantized_clone.py covers .clone() and torch.clone() across the four formats plus both HybridQuantizedTensor storage directions, then mutates the clone and checks the original buffers directly where dequantization is unavailable. The remaining gate is the repository GPU L0 run on this head.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants