Repository navigation
[PyT] Add context-parallel attention benchmark results - #3582
sudhakarsingh27 wants to merge 12 commits into
Conversation
Add reproducible BF16/FP16 CP workloads, mean-of-iteration-rank-max endpoint timing, cross-backend checks, and phase-level NVTX ranges. Keeping benchmark setup and result reduction outside timed samples makes endpoint measurements comparable while preserving the existing correctness and FP8 paths. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Replace the earlier result notes with a concise measured-data report and seven supporting figures. Keeping the publication to a report, snapshots, and a direct benchmark/profile example makes the review self-contained without carrying intermediate analysis tables. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
7da58be to
5d4673d
Compare
sudhakarsingh27
left a comment
There was a problem hiding this comment.
- Mention TE, cuDNN, pytorch version
- Remove the "matching profile"
- Talk about second best option as well
Preserve the independent non-CP reference check introduced upstream while keeping benchmark-only execution outside the correctness path. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Publish the refreshed measured-data report and its referenced snapshots so readers can compare both backend latencies without inferring absolute times from normalized plots. Keep the endpoint values alongside figures for reproducibility. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
[Low impact] The reviewed changes appear safe to merge, with no new actionable issues found. SummaryAdds context-parallel attention workloads, structured timing results, and an H100/B200 selection report.
Reviews (4) · Last reviewed commit: "Remove the benchmark README self-referen..." · Reviewed by Greptile |
| @@ -1,222 +1,177 @@ | |||
| # Choosing Context-Parallel Attention Paths from Measured Data | |||
There was a problem hiding this comment.
- trim .csv file
- remove scaling section in README because we're not sure about the conclusion
- a moving profile for fused/flash; will update as time goes on
There was a problem hiding this comment.
Trimmed the CSV to nine columns without changing any measured values. Removed the standalone CP-scaling and H100/B200 ratio sections. The README now summarizes how to choose a backend and communication type, including alternatives when A2A is unavailable.
| @@ -0,0 +1,426 @@ | |||
| # Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
The revisions we checked merged cleanly. We haven’t GPU-tested the two PRs together.
| a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True | ||
| ) | ||
| name = a2a_input_names[i - 1] | ||
| with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.launch"): |
There was a problem hiding this comment.
Let's remove these sub-ranges. Thanks!
There was a problem hiding this comment.
Removed the PR-added CP attention ranges in ff1aa311. Standalone benchmark profiler controls are unchanged.
Dense benchmark inputs are allocated per rank, so they bypass the reference path that used to supply the bias chunk indices. Share the dense chunk mapping between both paths and skip indexing broadcast bias. Cover both bias layouts without bringing back full-sequence allocations. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Drop the CP scaling and cross-system ratio sections and their obsolete assets so the report focuses on backend and communication selection. Keep the exact measured latencies and the fields needed to reproduce the figures; document shared settings once in the README. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Explain the measured backend crossover and keep communication-type comparisons scoped to the selected backend. Make pure all-to-all eligibility explicit so readers can identify applicable configurations. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Give readers a concise way to apply the benchmark findings to A2A eligibility, causal backend crossovers, and sliding-window communication choices. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the benchmark runner and timing controls without adding fine-grained instrumentation to production context-parallel attention. The attention implementation reverts to the merged upstream version. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the benchmark example self-contained without linking the report back to the pull request that introduces it. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Description
The benchmark path deliberately excludes FP8 and CUDA graph changes. Existing correctness and FP8 behavior remain available outside benchmark mode.
Validation
Distributed THD GPU tests were not rerun on the current validation host because they require sm90 or newer. CI is pending after the latest scope reduction.
Checklist
main