Skip to content

Enable hybrid CUDA IPC and EFA GPU exchange - #413

Draft
devavret wants to merge 1 commit into
devavret/g7e48-local-nvme-ucx-q18from
devavret/g7e48-hybrid-cuda-ipc-srd-followup
Draft

Enable hybrid CUDA IPC and EFA GPU exchange#413
devavret wants to merge 1 commit into
devavret/g7e48-local-nvme-ucx-q18from
devavret/g7e48-hybrid-cuda-ipc-srd-followup

Conversation

@devavret

@devavret devavret commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • backport OpenUCX PR #11865 on top of the UCX 1.22 worker build so RTX Pro 6000 Blackwell gets a realistic CUDA IPC bandwidth estimate
  • enable sm and cuda_ipc alongside EFA/SRD so UCX can use PCIe P2P for reachable neighboring GPUs and SRD for the remaining pairs
  • default UCX_MAX_RNDV_RAILS to 2 because it was the better of the two hybrid settings in the complete-suite A/B

This is a draft stacked on #412. The UCX patch should be removed once the upstream change is available in the UCX release used by the worker image.

Validation

All query tests used local-NVMe SF3K, 8 GPU workers, 2 drivers, 3 iterations, and no exchange compression. Streaming aggregation was enabled only as an experimental config change and is not part of this PR.

Complete TPC-H suite Pure SRD Hybrid, rails 1 Hybrid, rails 2
Sum of hot-query averages 57.302 s 58.070 s 57.813 s
Change from pure SRD 1.34% slower 0.89% slower
Q9 hot average 9.108 s 9.574 s 9.481 s
Q18 hot average 4.432 s 4.556 s 4.465 s
  • All 22 queries executed in all three arms.
  • Rails 2 was 0.44% faster than rails 1, but neither hybrid configuration improved the complete suite over pure SRD.
  • Validation passed for 21 queries. Q15 has a known float-data MAX(total_revenue) equality instability: pure SRD and rails 1 returned no row, while rails 2 returned one row.
  • A CUDA-memory UCX pair test changed GPU 0 to GPU 1 from SRD at 30.29 GB/s to CUDA IPC at 52.41 GB/s; GPU 0 to GPU 2 remained on SRD as expected from the P2P matrix.

This remains a draft because the transport-selection mechanism works, but the full-suite data does not yet demonstrate an end-to-end performance win.

@copy-pr-bot

copy-pr-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant