[xpupti] Don't emit ac2g flow endpoints on XPU_DRIVER records by aostrowski-hbn · Pull Request #1549 · pytorch/kineto

aostrowski-hbn · 2026-08-26T15:00:46Z

What

Stop the XPU (xpupti) profiler plugin from attaching a CPU→GPU ("ac2g") flow
endpoint to XPU_DRIVER (Level Zero ze*) records.

Why

The SYCL runtime record, the Level Zero driver (ze*) record and the device
kernel all share one PTI correlation id, and
handleRuntimeKernelMemcpyMemsetActivities set flow.id/flow.type on all of
them — so the driver record became an extra ac2g flow endpoint. In the exported
Chrome/Perfetto trace this draws a redundant host→host arrow from the
runtime "submit" slice to its own nested ze* child (already its child on the
same track), and the non-launch ze* calls — which get fresh correlation ids —
emit dangling flow finishes with no matching start. ac2g means async
CPU→GPU: a flow should link a host launch to the device op, never both start
and end on the host.

How

A single ac2gFlowRole() helper maps the record's activity type to
{None, Source, Destination}, and flow.id, flow.type and flow.start are
set together from it. The host runtime record is the Source, kernel / memcpy /
memset are Destinations, and XPU_DRIVER records get None — so their flow
id stays 0 and output_json's flowId() > 0 guard emits no link. The ze*
slices themselves stay in the trace; only their redundant arrows go.

Naming the flow type in the predicate is deliberate rather than provisional:
ac2g is the only flow this plugin can emit — fwdbwd links are generated by
the PyTorch CPU-side profiler (generateForwardBackwardLink) from autograd
sequence numbers and never reach a device backend. Source/Destination also
says which end a record is, which the reviewed carriesFlow name did not.
Folding the two predicates (startsFlow + carriesFlow) into one removes a
second list of activity types that could drift out of sync with the first.

Test

XpuptiActivityHandlersTest asserts the observable flow fields on hand-built
PTI records rather than re-asserting the predicate's own table: the runtime
record is a flow start, kernel / memcpy / memset are flow ends, and the
XPU_DRIVER record carries neither a flow id nor a flow type — an id with
type 0 would reach handleGenericLink, which then logs "Unknown flow type" for
every such record.

Built and the full xpupti ctest suite run on Intel GPU hardware (Intel® Arc™
Pro B-Series Graphics and Intel Data Center GPU Max Series); this revision was
re-verified on Intel Data Center GPU Max Series against PTI 1.2.0 — 22/22 pass
(XpuptiScopeProfilerTest.PerKernelScope is skipped upstream, #1533).
Before/after Perfetto traces confirm the redundant driver arrows drop to zero
while the runtime→kernel arrows are unchanged.

A companion PyTorch profiler test (kept in draft until this lands) guards the
same property end-to-end.

Companion (draft): pytorch/pytorch#194904

The SYCL runtime record, the Level Zero driver (ze*) record and the device kernel share one correlation id, and handleRuntimeKernelMemcpyMemsetActivities sets flow.id/flow.type on all of them. The driver record then becomes an extra ac2g flow endpoint, so Perfetto draws a redundant host->host arrow from the runtime "submit" slice to its nested ze* child; non-launch ze* calls (which get fresh correlation ids) also emit dangling flow finishes with no matching start. Gate the flow assignment on a new carriesFlow() predicate so only the runtime (flow source) and the device activities (kernel/memcpy/memset, flow destination) carry the ac2g flow; XPU_DRIVER records carry none. Add predicate unit tests and a handler-level test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

aostrowski-hbn · 2026-08-26T15:05:27Z

@pytorchbot label 'ciflow/xpu'

pytorch-bot · 2026-08-26T15:06:45Z

The ciflow label(s) ciflow/xpu will be added, but CI won't be triggered until the workflows are approved (scroll to the bottom of this page).

Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2026-08-26T15:06:49Z

~~Workflows were awaiting approval.~~ CI has now been triggered for the ciflow labels on this PR.

aostrowski-hbn · 2026-08-26T15:08:16Z

@guangyey, @gujinghui, @EikanWang please review

Review feedback on pytorch#1549: `carriesFlow` claimed to test whether a record is a "CPU->GPU flow endpoint", yet returned true for XPU_RUNTIME -- the host side, which is the flow *source*, not its destination. The name also did not say which of the two flow types (fwdbwd, ac2g) it meant. Replace `startsFlow` and `carriesFlow` with a single `ac2gFlowRole()` returning {None, Source, Destination}: the flow type is named in the predicate, `Source`/`Destination` replaces the ambiguous "endpoint", and one list of activity types can no longer drift out of sync with a second one -- flow id, type and start are now set together in one place. Scoping the name to ac2g is accurate rather than provisional: the plugin never emits fwdbwd, which the PyTorch CPU-side profiler generates and no device backend sees. Both predicates were public static members only so the unit test could reach them, and a test that re-asserts the switch's own table proves little. Drop XpuptiFlowCorrelationTest, move the function into an anonymous namespace in the only translation unit that uses it, and assert the observable behaviour in XpuptiActivityHandlersTest instead: memcpy and memset records are flow destinations, and the driver record's flow *type* stays 0 along with its id (an id with type 0 reaches handleGenericLink, which then logs "Unknown flow type" for every such record).

…o-driver-flow-arrows

An activity selection that keeps XPU_DRIVER but drops XPU_RUNTIME lost every CPU->GPU arrow, as only a runtime record could start a flow. Let the driver record take that role when the runtime view is absent; it shares the device work's correlation id. While both views are collected the driver record stays out of the flow: an arrow to a span nested under the submit would run host->host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…o-driver-flow-arrows # Conflicts: # libkineto/src/plugin/xpupti/XpuptiActivityProfilerSession.h # libkineto/test/xpupti/XpuptiActivityHandlersTest.cpp

…iver-flow-arrows' into dev/adrianos/xpupti-no-driver-flow-arrows # Conflicts: # libkineto/src/plugin/xpupti/XpuptiActivityHandlers.cpp

The session borrowed a reference into a ConfigDerivedState that kineto rebuilds on every configure(), and it now also derives a bitmask from that set. Copying it removes both the dangling-reference window and the second representation that could drift from the first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…o-driver-flow-arrows

The ac2g comment now says what the function decides instead of how the records nest, and stays with the XPU_RUNTIME/XPU_DRIVER names the rest of Kineto uses. The mask's default constructor had no caller, and the local enum has no reason to fix its underlying type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The session held the selection twice: the caller's set and a bitmask derived from it, which could drift apart. Our own enable/disable API now takes the mask, so the session keeps only that. The selection is fixed for a session's lifetime, since kineto clears its sessions before it rebuilds the config state on the next configure(). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

An unconstrained template parameter accepts anything and reports the mismatch from deep inside the body; std::invocable states the expected signature at the boundary, where the caller sees it. Kineto already uses this form in TypedMetadata.h. The visiting order is now part of the contract, so tests cover it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

aostrowski-hbn · 2026-09-10T08:28:43Z

@ryanzhang22 @scotts please review

aostrowski-hbn · 2026-09-11T07:47:48Z

The CUDA failure is unrelated and seen also in other PRs (#1562, #1561, #1558).

aostrowski-hbn · 2026-09-15T09:56:23Z

@ryanzhang22 @scotts can you take a look? The unrelated CI failure should not block this PR from being merged.

ryanzhang22 · 2026-09-15T14:14:18Z

Sorry was out on vacation for a few days. Will take a look.

ryanzhang22 · 2026-09-15T19:01:28Z

-bool XpuptiActivityProfilerSession::startsFlow(ActivityType activityType) {
-  // Only host runtime records start the CPU->GPU flow. The runtime view is
-  // already filtered to work-submitting APIs via
-  // ptiViewEnableRuntimeApiClass(PTI_API_CLASS_GPU_OPERATION_CORE). Driver


In XPU, is there an expectation that all driver calls produce a kernel event? I see that PTI_API_CLASS_GPU_OPERATION_CORE is used for filtering of runtime events, do you need the same thing for driver events?

JFYI for CUDA jobs we check if we need a flow arrow on the API call level: https://github.com/pytorch/kineto/blob/main/libkineto/src/CuptiCbidRegistry.cpp#L87

This lets us filter out individual calls (but we don't do the runtime -> driver fallback).

Not all driver calls produce kernels, but once we have filtering in place #1570 virtually all of them will (except barriers, I think).

The user might still want to filter out driver events.

The driver record's role depends on whether the runtime view is traced, which a free function could only learn through a bool argument set at the call site -- far from the one case that reads it. As a member of the session that owns the activity selection, it consults that selection in the XPU_DRIVER branch itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Enabling and disabling the views repeated the same list of activity types, so a type added to one could be missed in the other. A single mapping now answers which view a type is collected through, leaving enable with just the extras a view needs on top. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ryanzhang22 · 2026-09-23T15:18:20Z

@aostrowski-hbn Could you guys take a look at the XPU CI failure? Seems like it's happening on other PRs too.

Last one! Includes the following commits: - [Kineto] Exclude test on windows (pytorch/kineto#1575) 638a3ef - Opt-in per-thread activity buffers on the ROCm HIP path (pytorch/kineto#1573) 5dc9333 - Parse DISABLE_CUPTI_LAZY_REINIT by value (pytorch/kineto#1568) f140a13 - [Kineto] Add lookback window and sampling interval as config options (pytorch/kineto#1561) fcda441 - [xpupti] Don't emit ac2g flow endpoints on XPU_DRIVER records (pytorch/kineto#1549) f0df071 Pull Request resolved: #198560 Approved by: https://github.com/sanrise Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

aostrowski-hbn mentioned this pull request Aug 26, 2026

[profiler][xpu] Test that ac2g CPU->GPU flow arrows end on the device pytorch/pytorch#194904

Draft

meta-cla Bot added the cla signed label Aug 26, 2026

pytorch-bot Bot added the ciflow/xpu label Aug 26, 2026

PawelSwider2000 approved these changes Aug 27, 2026

View reviewed changes

tsocha reviewed Aug 27, 2026

View reviewed changes

Comment thread libkineto/src/plugin/xpupti/XpuptiActivityProfilerSession.h Outdated

aostrowski-hbn added 2 commits August 31, 2026 21:04

Merge remote-tracking branch 'origin/main' into dev/adrianos/xpupti-n…

e4db789

…o-driver-flow-arrows

aostrowski-hbn requested a review from tsocha September 1, 2026 07:53

tsocha reviewed Sep 1, 2026

View reviewed changes

Comment thread libkineto/src/plugin/xpupti/XpuptiActivityHandlers.cpp Outdated

Fix review comment

1af54a0

tsocha approved these changes Sep 1, 2026

View reviewed changes

aostrowski-hbn marked this pull request as draft September 1, 2026 11:02

gujinghui reviewed Sep 2, 2026

View reviewed changes

Comment thread libkineto/src/plugin/xpupti/XpuptiActivityHandlers.cpp

aostrowski-hbn and others added 8 commits September 7, 2026 17:14

Merge remote-tracking branch 'origin/main' into dev/adrianos/xpupti-n…

263f135

…o-driver-flow-arrows # Conflicts: # libkineto/src/plugin/xpupti/XpuptiActivityProfilerSession.h # libkineto/test/xpupti/XpuptiActivityHandlersTest.cpp

Merge remote-tracking branch 'intel-staging/dev/adrianos/xpupti-no-dr…

fbb7270

…iver-flow-arrows' into dev/adrianos/xpupti-no-driver-flow-arrows # Conflicts: # libkineto/src/plugin/xpupti/XpuptiActivityHandlers.cpp

Merge remote-tracking branch 'origin/main' into dev/adrianos/xpupti-n…

c6eae01

…o-driver-flow-arrows

aostrowski-hbn requested review from PawelSwider2000, gujinghui and tsocha September 9, 2026 12:57

gujinghui approved these changes Sep 10, 2026

View reviewed changes

aostrowski-hbn marked this pull request as ready for review September 10, 2026 08:27

ryanzhang22 reviewed Sep 15, 2026

View reviewed changes

tsocha reviewed Sep 21, 2026

View reviewed changes

aostrowski-hbn and others added 3 commits September 22, 2026 12:17

[xpupti] Rename the mask index helper to asIndex

2c0c691

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

aostrowski-hbn requested review from gujinghui and tsocha September 22, 2026 13:17

tsocha approved these changes Sep 22, 2026

View reviewed changes

ryanzhang22 approved these changes Sep 23, 2026

View reviewed changes

ryanzhang22 merged commit f0df071 into pytorch:main Sep 23, 2026
11 of 13 checks passed

aostrowski-hbn deleted the dev/adrianos/xpupti-no-driver-flow-arrows branch September 24, 2026 08:10

This was referenced Sep 24, 2026

Update third_party/kineto submodule to 5dc9333 pytorch/pytorch#198552

Closed

Update third_party/kineto submodule to 638a3ef pytorch/pytorch#198560

Closed

Conversation

aostrowski-hbn commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Test

Uh oh!

aostrowski-hbn commented Aug 26, 2026

Uh oh!

pytorch-bot Bot commented Aug 26, 2026

Uh oh!

pytorch-bot Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aostrowski-hbn commented Aug 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aostrowski-hbn commented Sep 10, 2026

Uh oh!

aostrowski-hbn commented Sep 11, 2026

Uh oh!

aostrowski-hbn commented Sep 15, 2026

Uh oh!

ryanzhang22 commented Sep 15, 2026

Uh oh!

ryanzhang22 Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

aostrowski-hbn Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanzhang22 commented Sep 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aostrowski-hbn commented Aug 26, 2026 •

edited

Loading

pytorch-bot Bot commented Aug 26, 2026 •

edited

Loading