Add activation-aware local Hessian scale calibration

[realAsma]

What does this PR do?

Type of change: new feature

Extends local_hessian weight-scale calibration with an opt-in activation-quantization-aware block-wise output MSE objective.

When act_quant_aware=True, scale search minimizes the exact expanded error for ||X_q W_q - X W_0||² using:

• H = X_qᵀ X_q / B
• P = X_qᵀ (X_q - X) / B
• the quadratic Hessian term plus the activation-error cross term

The default remains False, preserving the existing local_hessian objective. The reference path and Triton FP8 scale-sweep fast path both support the cross term. Dense and fused-MoE activation capture are covered, with explicit fallback for transformed input quantizers.

For both regular and activation-aware local_hessian, use batch_size=1 and layerwise=True for calibration fidelity and bounded memory.

Usage

quant_cfg["algorithm"] = {
    "method": "local_hessian",
    "act_quant_aware": True,
    "fp8_scale_sweep": True,
}

The new nvfp4_local_hessian_act_aware recipe alias enables this configuration directly.

Testing

• tests/unit/torch/quantization/test_local_hessian.py: 16 passed
• Fused-MoE local-Hessian focused cases: 2 passed
• tests/unit/recipe/test_presets.py: 6 passed
• Activation-aware Triton/reference parity, validation, and end-to-end GPU cases: 5 passed
• Existing Hessian Triton/reference parity matrix: 18 passed
• Pre-commit on all changed files: passed

Before your PR is "Ready for review"

• Is this change backward compatible?: ✅
• If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A
• Did you write any new necessary tests?: ✅
• Did you update Changelog?: ❌ — pending while this PR is a draft
• Did you get Claude approval on this PR?: ❌ — pending while this PR is a draft

Additional Information

The implementation follows the activation-quantization-aware block-wise output MSE derivation supplied with the feature request.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3a5a45cb-8820-47c6-9784-c6c3eb6dfaae

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch asma/activation-quant-aware-scale-setting

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA.github.io/Model-Optimizer/pr-preview/pr-1976/
Built to branch gh-pages at 2026-07-14 23:55 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.33%. Comparing base (f479e78) to head (fd8f9d9).

Files with missing lines	Patch %	Lines
...torch/kernels/quantization/gemm/nvfp4_fp8_sweep.py	0.00%	7 Missing ⚠️
modelopt/torch/quantization/model_calib.py	96.22%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1976   +/-   ##
=======================================
  Coverage   76.33%   76.33%           
=======================================
  Files         522      522           
  Lines       58452    58505   +53     
=======================================
+ Hits        44618    44662   +44     
- Misses      13834    13843    +9     

Flag	Coverage Δ
unit	55.40% <85.71%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.