Skip to content

DirectML EP: add Tier-2 op+activation fusion, fp16 exec flags, and ORT parity fixes - #52

Merged
apwojcik merged 2 commits into
mainfrom
activation_fusions
Jul 22, 2026
Merged

DirectML EP: add Tier-2 op+activation fusion, fp16 exec flags, and ORT parity fixes#52
apwojcik merged 2 commits into
mainfrom
activation_fusions

Conversation

@KenLagos

@KenLagos KenLagos commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Add a generic Tier-2 BaseOp+Activation fusion rule and align several DirectML plugin behaviors with ONNX Runtime's DirectML execution provider.

Op+Activation fusion (new):

  • Add op_activation_ep_fusion.{h,cc} and fusion_activation_helper.h, fusing an optional downstream activation into Add, Sum, Gemm, Conv, ConvTranspose, BatchNormalization, InstanceNormalization, and MeanVarianceNormalization via the DML operator's FusedActivation field, with allowed activations filtered per op and per device to match ORT's fusability tables.
  • Fix a bug where the eager compile path never recorded compiled_slot_sizes, causing the fused kernel to be needlessly recompiled on every inference instead of once (~2x perf regression on fused Conv/ConvTranspose models).

Runtime (non-constant) weight support:

  • Bind Conv/ConvTranspose/BatchNorm/InstanceNorm weight, scale, bias, mean, and variance inputs per slot, either directly from a runtime OrtValue or via a one-time static upload for true constants, and defer kernel compilation until all weight shapes are known.

ORT parity fixes:

  • BatchNormalization epsilon now defaults to 0.0 (InstanceNormalization keeps 1e-5), matching ORT's per-op defaults.
  • MeanVarianceNormalization explicit axes are normalized for negative values, matching ORT's GetDmlAdjustedAxis.

fp16 execution flags:

  • Set DML_EXECUTION_FLAG_ALLOW_HALF_PRECISION_COMPUTATION when a graph is entirely fp16, in full-graph/per-partition compilation and in the QuickGelu, BiasGelu, and FusedMatMul fusion kernels.

Added a follow up

Fix Tier-0 graph fusion binding validation failure for ConvInteger models
The owned_graph_input_indices construction used raw packed DML slot
indices into the ONNX-ordered input_names array, causing initializer
weights to be looked up under wrong names for operators with
input_name_reorder (e.g. ConvInteger). This left OWNED_BY_DML
initializers unbound in the init binding table, triggering a DirectML
BindingValidator E_INVALIDARG and device removal during session init.

Additional fixes:

  • Resolve passthrough aliases in consumed_initializer_names tracking
    so initializers consumed through renamed edges get DML graph input slots
  • Use min(actualTensorBytes, totalTensorSizeInBytes) for constant node
    DataSize to avoid reading past ORT tensor allocations
  • Add dml_input_slot_indices to ConvInteger, QLinearConv, MatMulInteger,
    QLinearMatMul, and MatMulIntegerToFloat for correct edge wiring when
    optional inputs are absent
  • Fix filter zero-point and scale axis placement from 0 to 1 (C-axis)
    in ConvInteger and QLinearConv per DML spec
  • Guard InitializeOperator/Flush with ORT_TRY/ORT_CATCH so a single
    bad partition falls back instead of aborting the process
  • Extract DrainDmlDebugMessages helper and add device-removed HRESULT
    logging in CloseAndExecute
  • Rename ambiguous variables (cn, sg, im, npi, ni, ib, ci) for
    readability

…T parity fixes

Add a generic Tier-2 BaseOp+Activation fusion rule and align several
DirectML plugin behaviors with ONNX Runtime's DirectML execution provider.

Op+Activation fusion (new):
- Add op_activation_ep_fusion.{h,cc} and fusion_activation_helper.h,
  fusing an optional downstream activation into Add, Sum, Gemm, Conv,
  ConvTranspose, BatchNormalization, InstanceNormalization, and
  MeanVarianceNormalization via the DML operator's FusedActivation field,
  with allowed activations filtered per op and per device to match ORT's
  fusability tables.
- Fix a bug where the eager compile path never recorded compiled_slot_sizes,
  causing the fused kernel to be needlessly recompiled on every inference
  instead of once (~2x perf regression on fused Conv/ConvTranspose models).

Runtime (non-constant) weight support:
- Bind Conv/ConvTranspose/BatchNorm/InstanceNorm weight, scale, bias,
  mean, and variance inputs per slot, either directly from a runtime
  OrtValue or via a one-time static upload for true constants, and defer
  kernel compilation until all weight shapes are known.

ORT parity fixes:
- BatchNormalization epsilon now defaults to 0.0 (InstanceNormalization
  keeps 1e-5), matching ORT's per-op defaults.
- MeanVarianceNormalization explicit axes are normalized for negative
  values, matching ORT's GetDmlAdjustedAxis.

fp16 execution flags:
- Set DML_EXECUTION_FLAG_ALLOW_HALF_PRECISION_COMPUTATION when a graph is
  entirely fp16, in full-graph/per-partition compilation and in the
  QuickGelu, BiasGelu, and FusedMatMul fusion kernels.
@KenLagos
KenLagos requested review from apwojcik and tperry-amd July 21, 2026 14:39
@KenLagos KenLagos self-assigned this Jul 21, 2026
tperry-amd
tperry-amd previously approved these changes Jul 21, 2026
…dels

The owned_graph_input_indices construction used raw packed DML slot
indices into the ONNX-ordered input_names array, causing initializer
weights to be looked up under wrong names for operators with
input_name_reorder (e.g. ConvInteger). This left OWNED_BY_DML
initializers unbound in the init binding table, triggering a DirectML
BindingValidator E_INVALIDARG and device removal during session init.

Additional fixes:
- Resolve passthrough aliases in consumed_initializer_names tracking
  so initializers consumed through renamed edges get DML graph input slots
- Use min(actualTensorBytes, totalTensorSizeInBytes) for constant node
  DataSize to avoid reading past ORT tensor allocations
- Add dml_input_slot_indices to ConvInteger, QLinearConv, MatMulInteger,
  QLinearMatMul, and MatMulIntegerToFloat for correct edge wiring when
  optional inputs are absent
- Fix filter zero-point and scale axis placement from 0 to 1 (C-axis)
  in ConvInteger and QLinearConv per DML spec
- Guard InitializeOperator/Flush with ORT_TRY/ORT_CATCH so a single
  bad partition falls back instead of aborting the process
- Extract DrainDmlDebugMessages helper and add device-removed HRESULT
  logging in CloseAndExecute
- Rename ambiguous variables (cn, sg, im, npi, ni, ib, ci) for
  readability
@apwojcik
apwojcik merged commit a501556 into main Jul 22, 2026
1 check passed
@apwojcik
apwojcik deleted the activation_fusions branch July 22, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants