warnings.warn(f"RealQuantLinear: No real-quant GEMM found: {self}.")

[zewenli98]

Describe the bug

I followed this quant recipe to quantize Alpamayo 1.5 to FP8 on B300. First, I ran the quantize.py to quantize Alpamayo 1.5 to FP8 and save the FP8 model (mtq.compress() is called before saving). It was successfully saved. Then, when I ran the eval.py to test the FP8 model, I got the error:

.../lib/python3.12/site-packages/modelopt/torch/quantization/nn/modules/quant_linear.py:214: UserWarning: RealQuantLinear: No real-quant GEMM found: RealQuantQuantLinear(
  in_features=2048, out_features=2, bias=True
  (input_quantizer): TensorQuantizer((4, 3) bit fake per-tensor amax=2.7363 calibrator=MaxCalibrator quant)
  (output_quantizer): TensorQuantizer(disabled)
  (weight_quantizer): TensorQuantizer((4, 3) bit per-tensor amax=0.1025 calibrator=MaxCalibrator quant)
).
  warnings.warn(f"RealQuantLinear: No real-quant GEMM found: {self}.")

I only tested on B300, so I'm not sure if that's because Blackwell is not well supported or it's a general issue.

Steps/Code to reproduce bug

Please follow https://github.com/NVlabs/alpamayo-recipes/tree/main/recipes/alpamayo1_5_quant#readme to install ENV. and then run:

# quant + save model
python quantize.py --quant_format=fp8 --num_of_calib_clips=10 --save_model_dir=./outputs

# eval
python eval.py --ckpt ./outputs/alpamayo1.5_fp8_calib10
# then the error pops up

Expected behavior

The eval script should run in FP8 without error.

Who can help?

• ?

System information

• Container used (if applicable): nvcr.io/nvidia/pytorch:26.04-py3
• OS (e.g., Ubuntu 22.04, CentOS 7, Windows 10): ? Ubuntu 22.04
• CPU architecture (x86_64, aarch64): x86_64
• GPU name (e.g. H100, A100, L40S): B300
• GPU memory size: ?
• Number of GPUs: ?
• Library versions (if applicable):
  • Python: 3.12
  • ModelOpt version or commit hash: 0.43
  • CUDA: 13.2
  • PyTorch: 2.12.1
  • Transformers: ?
  • TensorRT-LLM: ?
  • ONNXRuntime: ?
  • TensorRT: ?
• Any other details that may help: ?

[h-guo18]

Hi @cjluo-nv , could you take a look at this bug. Thanks

[rohansjoshi]

You shared a warning, not an error? Could you share the full log?

We have a working example here for Alpamayo-1 quantization, and the exported checkpoint can be evaluated with eval.py.