Repository navigation
Conversation
Signed-off-by: ZenAlexa <zimingwang945@gmail.com>
[Medium impact] The PR appears safe to merge based on the code reviewed; current-head GPU tests remain to be run. SummaryAdds opt-in CPU export for
Both earlier findings are addressed: distributed checks now run under Reviews (10) · Last reviewed commit: "[PyTorch] Merge current main" · Reviewed by Greptile |
Signed-off-by: ZenAlexa <zimingwang945@gmail.com>
Signed-off-by: ZenAlexa <zimingwang945@gmail.com>
Signed-off-by: ZenAlexa <zimingwang945@gmail.com>
Signed-off-by: ZenAlexa <zimingwang945@gmail.com>
Signed-off-by: ZenAlexa <zimingwang945@gmail.com>
Signed-off-by: ZenAlexa <zimingwang945@gmail.com>
Signed-off-by: Ziming Wang <zimingwang945@gmail.com>
Signed-off-by: Ziming Wang <zimingwang945@gmail.com>
|
Rechecked the current head |
November 16 mainline integration
Revision
3c97ed2a07e0includesmain@4f5c08e48b3ethrough a normal merge. The contribution source and regression files retain their pre-integration contents.Runtime and hardware results below retain their named revisions. This synchronization adds source and consumer-delta review, and current-head runtime/device CI remains requested from the project.
Description
Add opt-in
state_dict_cpu_offload=FalsetoFusedAdam, allowing checkpoint callers that use the no-argumentstate_dict()API to move each converted state to CPU before exporting the next one. This addresses the GPU expansion peak reported in Automodel #4209, where checkpoint CPU offload runs after the complete optimizer dictionary has already been materialized.Fixes #3668.
Type of change
Changes
Keep the existing state conversion and DTensor wrapping, then copy each export state to CPU when the option is enabled. Placing the copy after wrapping avoids CUDA meshes moving a CPU local tensor back to the GPU. Exported values, dtypes, keys and distributed metadata remain unchanged, while live optimizer state and parameter-group metadata retain their existing ownership and placement.
Add a standalone lifecycle scenario to the existing L0 runner, covering repeated exports, restoration and continued updates, including sharded DCP with the
flatten_optimizer_state_dict=Trueconfiguration used by Automodel. The option is a constructor setting so the existing optimizer factory can forward it without changing DCP's call signature.Verification
After merging main
925832d7on November 17, 14 single-GPU export, restore, and continued-step scenarios passed. The one-parameter and eight-parameter checks each reached a 1,048,576-byte temporary-allocation peak. These runs used the current Python source and official TE 2.21 extension on SM89; two-rank DTensor/DCP and a full current-main native build remain unrun.Merged main
63451d6b868athrough a normal merge atd080a53d1b75on November 1. The FusedAdam owner retains main'ste_device_typeimport migration, and the CPU export implementation and both QA entry points are unchanged. Current-head scoped pre-commit, repository license checks, Python syntax, QA shell syntax and contribution-diff checks pass. The merged revision has not been rebuilt or rerun on a GPU.Earlier native validation
The native build recorded for the
7a3c25482cd0contribution revision succeeded on H200 with PyTorch2.12.0.dev20260408+cu128, CUDA 12.8.61, GCC 13.3, cuDNN 9.20.0.48 and the source-pinned CUTLASS revision. The compiled common library, PyTorch extension and Python package are installed together as TE2.22.0.dev0, without a source overlay or platform-getter replacement. The earlier TE 2.20.2 compatibility run remains historical evidence.The distributed QA job now invokes the scenario with two ranks, closing the review finding about skipped DTensor/DCP checks. Its failure status goes through the job's existing
test_failpath.flatten=False/True × sync/asynccombinations. Full-state DCP also passes default/CPU export with synchronous and asynchronous saves, followed by load and continued updates.27selected FusedAdam regressions with12 deselected, using-k "TestFusedAdam and not MXFP8 and not Float8Block". The earlier main/candidate Python comparison passed the same selection for both arms.The GPU allocation increments measured in that earlier native run were:
These measurements use two BF16 moments and report temporary PyTorch allocated memory above the pre-export level. Comparing with a baseline that subsequently copies the complete dictionary to CPU gives the same peak differences and the same final CPU payload, with zero additional GPU export allocation retained at the common endpoint. Total process memory and checkpoint storage are separate quantities.
A full-state gather can move exported states back to the mesh device, as documented. The complete L0 matrix, minimum supported PyTorch version, larger cross-world-size configurations and original 64-GB200 recipe remain unrun. The FP8 scenarios emit the release's
Float8Tensor_to_copywarning and still pass their value and continued-update checks. Please run the project's full build and distributed optimizer matrix before merging.