Repository navigation
[PyTorch] Fix strict load_state_dict for te.ops.Linear without bias - #3915
bhargavchintam wants to merge 1 commit into
Conversation
Linear._load_from_state_dict added state_dict["bias"] = None when the op had no bias. torch leaves None parameters out of the keys it expects, so a strict load reported the added key as unexpected, and a bias-free te.ops.Linear could not load the state dict it had just saved. Add the bias key only when the op has a bias. Run TestCheckpointing.test_linear with bias=False as well as bias=True. Signed-off-by: Bindu Bhargava R Chintam <bindubhargavareddy@gmail.com>
[Medium impact] The PR appears safe to merge. SummaryFixes strict checkpoint loading for
No actionable issues found. Tests were not run during this review. Reviews (1) · Last reviewed commit: "[PyTorch] Fix strict load_state_dict for..." · Reviewed by Greptile |
lxcxjxhx
left a comment
There was a problem hiding this comment.
Looks right, and I checked the two things that usually make a fix of this shape incomplete.
Is the pattern repeated in the sibling ops? No. state_dict[...] = None inside _load_from_state_dict exists only in ops/linear.py:214 across transformer_engine/pytorch/ops/*.py, so fixing Linear alone is the whole fix rather than the first of several.
if self._bias_idx is not None and f"{prefix}bias" not in state_dict:
state_dict[f"{prefix}bias"] = state_dict[f"{prefix}basic_ops.{self._bias_idx}.bias"]Is dropping the None insertion safe for the reverse direction? A model with bias=False now leaves a checkpoint's bias key alone, so a bias-carrying checkpoint loaded into a bias-free op fails strict load as an unexpected key — which is the correct strict-load answer, and it is the same failure class the old code produced (the inserted None never matched either). Worth one sentence in the description so the next reader does not read it as a regression.
The parametrisation covers what the issue reports: bias × quantization × quantized_weight turns the single passing case into six, and the bias-free six are exactly the ones that failed on main before this change.
One thing I could not check here: the test needs a GPU and the compiled extension, so I did not run it — the reasoning above is from the diff plus a grep for the pattern.
|
Thanks for checking the sibling ops. I added the reverse case to the description. A |
|
@timmoon10, this is a two-line fix in |
Description
A bias-free
te.ops.Linearcan't load the state dict it just saved with a strictload_state_dict. It fails withUnexpected key(s) in state_dict: "0.bias".Linear._load_from_state_dictaddedstate_dict["bias"] = Nonewhen the op has no bias. torch leavesNoneparameters out of the keys it expects, so the added key was reported as unexpected. This PR adds the bias key only when the op has a bias.Loading a
bias=Truecheckpoint into a bias-free op still fails a strict load, as it should. With this change the unexpected keys are0.basic_ops.1.biasand0.basic_ops.1._extra_state; main also listed the inserted0.bias.Fixes #3914
Type of change
Changes
transformer_engine/pytorch/ops/linear.py:Linear._load_from_state_dictadds{prefix}biasonly when the op has a bias. Two lines replace five.tests/pytorch/test_fusible_ops.py:TestCheckpointing.test_linearis parametrized overbiasin(False, True)and passes it to bothte_ops.Linearconstructions.Testing
On an NVIDIA RTX PRO 6000 Blackwell (compute capability 12.0) in the NGC PyTorch 26.10 container (torch 2.15.0a0, CUDA 13.4), I built TE from main e6f9e86 with
NVTE_FRAMEWORK=pytorch NVTE_CUDA_ARCHS=120 python -m pip install -e . -v --no-build-isolation. I applied the test change first and the fix second, so both runs use the same build.python -m pytest -v tests/pytorch/test_fusible_ops.py -k "TestCheckpointing and test_linear"with main'sops/linear.pyand the new test cases:The first case again with
--tb=short:The same command on this branch (source tree
22b31bc8, the tree of commit 71968de):The 12 skipped cases are the
nvfp4,nvfp4_4over6andfp8_block_scalingrecipes, which skip on this GPU.The whole file on this branch, as
qa/L0_pytorch_unittest/test.shruns it (python -m pytest tests/pytorch/test_fusible_ops.py):The same 84 tests fail with main's
ops/linear.py(84 failed in 5.75s), so none of them comes from this change. They arefp8_delayed_scalingcases oftest_activation(36),test_layernorm_mlp(24),test_clamped_swiglu(16) andtest_swiglu(8). All of them stop incast_gated_tmawith "Only rowwise cast supported in this function" on this GPU.TE_PATH=$PWD PYTHON_ONLY=1 bash qa/L0_pytorch_lint/test.shon this branch:pre-commit run --from-ref origin/main --to-ref HEADon this branch:TE_PATH="$PWD" bash qa/L0_license/test.shexits 0. The MXFP8 recipe did not run: the test file adds it only whenmxfp8_availableis true, and it is false on this GPU.Checklist: