Skip to content

[PyT] Add context-parallel attention benchmark results - #3582

Open
sudhakarsingh27 wants to merge 12 commits into
NVIDIA:mainfrom
sudhakarsingh27:cp_bench
Open

sudhakarsingh27 wants to merge 12 commits into
NVIDIA:mainfrom
sudhakarsingh27:cp_bench

Conversation

@sudhakarsingh27

@sudhakarsingh27 sudhakarsingh27 commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Description

  • Add BF16/FP16 context-parallel benchmark workloads with rank-max endpoint timing and structured JSON output.
  • Add phase-level NVTX ranges and cross-backend checks for profile interpretation and correctness.
  • Publish a concise H100/B200 BF16 THD selection report with seven supporting figures and a direct benchmark/profile example.

The benchmark path deliberately excludes FP8 and CUDA graph changes. Existing correctness and FP8 behavior remain available outside benchmark mode.

Validation

  • Python compilation for all changed Python files
  • Repository license check
  • Black formatting hook
  • Report benchmark instructions and all seven figure links validated
  • Outgoing-content audit for local paths, hostnames, credentials, private issue references, and profiler locations

Distributed THD GPU tests were not rerun on the current validation host because they require sm90 or newer. CI is pending after the latest scope reduction.

Checklist

  • Changes are based on current main
  • Commits are signed off
  • Report documents the measured configurations, method, and limitations
  • No FP8 or CUDA graph benchmark changes are included
  • GPU CI passes

Add reproducible BF16/FP16 CP workloads, mean-of-iteration-rank-max endpoint timing, cross-backend checks, and phase-level NVTX ranges. Keeping benchmark setup and result reduction outside timed samples makes endpoint measurements comparable while preserving the existing correctness and FP8 paths.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Replace the earlier result notes with a concise measured-data report and seven supporting figures. Keeping the publication to a report, snapshots, and a direct benchmark/profile example makes the review self-contained without carrying intermediate analysis tables.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

@sudhakarsingh27 sudhakarsingh27 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Mention TE, cuDNN, pytorch version
  2. Remove the "matching profile"
  3. Talk about second best option as well

@KshitijLakhani KshitijLakhani changed the title Add context-parallel attention benchmark results [PyT] Add context-parallel attention benchmark results Oct 20, 2026
@KshitijLakhani
KshitijLakhani self-requested a review November 2, 2026 19:38
Preserve the independent non-CP reference check introduced upstream while
keeping benchmark-only execution outside the correctness path.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Publish the refreshed measured-data report and its referenced snapshots so
readers can compare both backend latencies without inferring absolute
times from normalized plots. Keep the endpoint values alongside figures
for reproducibility.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@cyanguwa
cyanguwa marked this pull request as ready for review November 18, 2026 18:08
@cyanguwa
cyanguwa self-requested a review as a code owner November 18, 2026 18:08
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The reviewed changes appear safe to merge, with no new actionable issues found.

Summary

Adds context-parallel attention workloads, structured timing results, and an H100/B200 selection report.

  • The attention runner reports context-parallel benchmark timings as JSON.
  • Benchmark workloads exercise long and uneven THD attention batches.
  • The report compares measured context-parallel attention choices on H100 and B200.

Reviews (4) · Last reviewed commit: "Remove the benchmark README self-referen..." · Reviewed by Greptile

Comment thread tests/pytorch/attention/benchmark_cp.py Resolved
Comment thread benchmarks/attention/README.md Outdated
@@ -1,222 +1,177 @@
# Choosing Context-Parallel Attention Paths from Measured Data

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. trim .csv file
  2. remove scaling section in README because we're not sure about the conclusion
  3. a moving profile for fused/flash; will update as time goes on

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trimmed the CSV to nine columns without changing any measured values. Removed the standalone CP-scaling and H100/B200 ratio sections. The README now summarizes how to choose a backend and communication type, including alternatives when A2A is unavailable.

@@ -0,0 +1,426 @@
# Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check if this works well with Alp's PR: #3408

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The revisions we checked merged cleanly. We haven’t GPU-tested the two PRs together.

a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True
)
name = a2a_input_names[i - 1]
with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.launch"):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove these sub-ranges. Thanks!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the PR-added CP attention ranges in ff1aa311. Standalone benchmark profiler controls are unchanged.

Dense benchmark inputs are allocated per rank, so they bypass the reference path that used to supply the bias chunk indices. Share the dense chunk mapping between both paths and skip indexing broadcast bias. Cover both bias layouts without bringing back full-sequence allocations.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Drop the CP scaling and cross-system ratio sections and their obsolete assets so the report focuses on backend and communication selection. Keep the exact measured latencies and the fields needed to reproduce the figures; document shared settings once in the README.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Explain the measured backend crossover and keep communication-type comparisons scoped to the selected backend. Make pure all-to-all eligibility explicit so readers can identify applicable configurations.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Give readers a concise way to apply the benchmark findings to A2A eligibility, causal backend crossovers, and sliding-window communication choices.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the benchmark runner and timing controls without adding fine-grained instrumentation to production context-parallel attention. The attention implementation reverts to the merged upstream version.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the benchmark example self-contained without linking the report back to the pull request that introduces it.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants