[None][perf] optimize encoder-decoder PyTorch performance by cascade812 · Pull Request #16706 · NVIDIA/TensorRT-LLM

cascade812 · 2026-07-22T04:19:05Z

Summary

accumulate replacement encoder work while decode continues, then release encoder microbatches at the configured iteration or token threshold
capture and replay encoder-forward CUDA graphs for encoder-decoder models using user-configured batch-size, packed-token, and sequence-length buckets
capture mixed decoder CUDA graphs for iterations containing newly admitted context requests and ongoing generation requests; enable this optimization by default when the encoder and decoder graph configurations produce usable shapes
prepare qualified mixed encoder/decoder batches in one native call using persistent host buffers
reuse request IDs and sequence slots for stable single-beam greedy decode batches, copy only compact argmax results to the host, and detect EOS/length completion without the general finish-reason tensor
preserve the general sampler path for draft decoding, beam search, log probabilities, sampling, biases, minimum length, stop words, bad words, and other excluded features

Why

BART continuous-admission serving repeatedly mixes replacement encoder requests with active decoder requests. The existing path launched small encoder batches, rebuilt mixed decoder metadata in Python, and reconstructed sampling/finish metadata on every greedy decode step. It also ran eligible encoder and mixed decoder work eagerly instead of replaying fixed-shape CUDA graphs.

These changes reduce per-iteration CPU launch and device-to-host overhead while keeping decoder generation active during encoder admission. The CUDA-graph capture layout is derived from the encoder runner's effective capture keys so padding and runtime replay use the same buckets.

User interface

For encoder-decoder models, encoder_cuda_graph_config=EncodeCudaGraphConfig(...) enables encoder-forward CUDA graphs and defines batch-size, total packed-token, and maximum sequence-length buckets. encoder_max_batch_size remains the hard encoder capacity and admission limit.

enable_encoder_decoder_mixed_cuda_graph controls the mixed-batch decoder optimization. It defaults to True but becomes effective only when both the encoder and decoder graph configurations produce usable capture shapes. Set it to False to retain separate encoder and decoder CUDA graphs while disabling mixed-batch graphs.

Performance

BART-large-CNN

This comparison uses the same deterministic natural-length CNN/DailyMail workload on both backends: 1,024 unique validation articles sampled without replacement using seed 0, encoder lengths of 93–1,022 tokens, and no truncation.

PyTorch improves throughput by 12.7–20.9% and reduces mean latency by 11.0–17.2% compared with legacy TensorRT.

Concurrency	PyTorch requests/s	Legacy TRT requests/s	PyTorch throughput gain	PyTorch mean latency	Legacy TRT mean latency	Mean reduction
8	65.570	54.257	+20.9%	121.259 ms	146.382 ms	17.2%
32	172.256	152.825	+12.7%	182.150 ms	204.649 ms	11.0%
64	251.797	221.568	+13.6%	246.155 ms	278.906 ms	11.7%

More BART workload, latency, distribution, and configuration details

Detailed latency percentiles

Concurrency	Backend	Mean	P50	P90	P99
8	PyTorch	121.259 ms	112.710 ms	190.480 ms	222.091 ms
8	Legacy TensorRT	146.382 ms	135.047 ms	241.317 ms	265.781 ms
32	PyTorch	182.150 ms	172.004 ms	280.996 ms	318.520 ms
32	Legacy TensorRT	204.649 ms	187.415 ms	336.094 ms	383.609 ms
64	PyTorch	246.155 ms	231.772 ms	388.755 ms	436.115 ms
64	Legacy TensorRT	278.906 ms	256.378 ms	458.957 ms	563.303 ms

Encoder input-length distribution

Lengths include the tokenizer's special tokens.

Count	Minimum	P10	P25	Median / P50	Mean	Population standard deviation	P75	P90	P95	P99	Maximum
1,024	93	330.3	447.0	636.0	626.487	217.948	800.0	924.7	979.85	1,011.0	1,022

Encoder tokens	1–128	129–256	257–384	385–512	513–640	641–768	769–896	897–1,024	Total
Requests	2	30	142	174	174	194	172	136	1,024
Percentage	0.195%	2.930%	13.867%	16.992%	16.992%	18.945%	16.797%	13.281%	100.000%

Configuration

PyTorch configuration:

NVIDIA H100 80 GB HBM3
TensorRT-LLM 1.3.0rc23, BF16, TRTLLM attention backend
maximum generated tokens: 128
encoder_max_batch_size=2 at concurrency 8 and 8 at concurrency 32/64
encoder graph batch sizes [1, 2] at concurrency 8 and [1, 2, 4, 8] at concurrency 32/64
encoder sequence-length buckets [512, 1024]

Legacy TensorRT configuration:

NVIDIA H100 80 GB HBM3
TensorRT-LLM 1.3.0rc21, BF16
maximum generated tokens: 127

Generated outputs were not bit-identical: PyTorch averaged approximately 73.37 output tokens per request, while legacy TensorRT averaged 72.32–72.41. Latency and throughput are end-to-end request measurements and are not normalized to identical output-token counts.

FLAN-T5 Large

This comparison uses google/flan-t5-large and a deterministic 1,024-request Super-NaturalInstructions workload derived from the official allenai/natural-instructions default/test split. Inputs longer than 512 tokens and reference outputs longer than 128 tokens are rejected rather than truncated.

With encoder and mixed encoder-decoder CUDA graphs enabled, PyTorch improves request throughput by 11.8–57.8% and reduces mean latency by 9.4–36.6% compared with legacy TensorRT.

Concurrency	PyTorch requests/s	Legacy TRT requests/s	PyTorch throughput gain	PyTorch mean latency	Legacy TRT mean latency	Mean reduction
8	173.724	110.082	+57.8%	45.788 ms	72.248 ms	36.6%
32	257.151	228.405	+12.6%	122.435 ms	137.233 ms	10.8%
64	335.378	300.102	+11.8%	185.578 ms	204.942 ms	9.4%

More T5 workload, latency, output-check, and configuration details

Each result is one clean run of 1,024 requests after an untimed warmup of one concurrency-sized request window. Timing covers closed-loop request submission through receipt of the final output.

Detailed throughput and latency measurements

Concurrency	Backend	Requests/s	Output tokens/s	Mean latency	P50	P90	P99
8	PyTorch	173.724	1,610.002	45.788 ms	33.588 ms	85.111 ms	155.986 ms
8	Legacy TensorRT	110.082	1,020.089	72.248 ms	39.165 ms	158.728 ms	322.818 ms
32	PyTorch	257.151	2,388.947	122.435 ms	91.386 ms	231.226 ms	435.339 ms
32	Legacy TensorRT	228.405	2,120.774	137.233 ms	75.573 ms	310.157 ms	630.880 ms
64	PyTorch	335.378	3,105.519	185.578 ms	132.623 ms	329.967 ms	616.529 ms
64	Legacy TensorRT	300.102	2,788.835	204.942 ms	121.228 ms	443.923 ms	927.274 ms

Output checks

Both APIs used greedy decoding with a maximum of 128 generated tokens. The benchmark normalizes the legacy decoder-start and EOS conventions before counting or hashing outputs.

Concurrency	Backend	Generated tokens	Average output length	Natural EOS	Length limit
8	PyTorch	9,490	9.268	1,021	3
8	Legacy TensorRT	9,489	9.267	1,021	3
32	PyTorch	9,513	9.290	1,021	3
32	Legacy TensorRT	9,508	9.285	1,021	3
64	PyTorch	9,482	9.260	1,021	3
64	Legacy TensorRT	9,516	9.293	1,021	3

An eight-request encoder-graph smoke test produced exactly the same greedy token sequences as eager execution. Full-run cross-backend token counts differ by at most 34 tokens (0.36%) because BF16 execution and batching change a small number of near-tie decoding decisions. This is a performance benchmark, not a task-accuracy evaluation.

Workload

Prompts use the following form:

Definition: <task definition>

Input: <instance input>

Output:

Selection is deterministic with seed 0:

tokenize with the google/flan-t5-large tokenizer
reject inputs longer than 512 tokens instead of truncating them
reject reference outputs longer than 128 tokens
select 256 requests from each encoder-length bucket using round-robin task sampling
shuffle the final 1,024 requests deterministically

The final workload covers 116 test tasks and 12 task categories. During selection, 7,086 candidate instances were rejected for exceeding 512 input tokens and five were rejected for exceeding 128 reference-output tokens.

Encoder input-length distribution

Count	Min	P10	P25	P50	Mean	Stddev	P75	P90	P95	P99	Max
1,024	33	54	64.75	128.5	169.332	116.033	256.25	341	400.85	484.77	510

Input tokens	1–64	65–128	129–256	257–512
Requests	256	256	256	256

Reference output-length distribution

Count	Min	P10	P25	P50	Mean	Stddev	P75	P90	P95	P99	Max
1,024	2	2	2	4	9.268	10.410	13.25	22	28	46.77	94

Category	Requests	Category	Requests
Title Generation	208	Question Rewriting	154
Textual Entailment	137	Answerability Classification	114
Coreference Resolution	105	Dialogue Act Recognition	85
Grammar Error Correction	55	Keyword Tagging	51
Data to Text	46	Word Analogy	29
Cause Effect Classification	29	Overlap Extraction	11

Configuration

Item	Value
GPU	NVIDIA H100 80 GB HBM3
PyTorch-path TensorRT-LLM	1.3.0rc23, BF16, TRTLLM attention backend
Legacy TensorRT-LLM	1.3.0rc21, BF16
Model	`google/flan-t5-large`, TP=1, PP=1
Generation	Greedy, natural EOS, maximum 128 generated tokens
Traffic	Closed loop, fixed concurrency 8/32/64, 1,024 requests

PyTorch decoder CUDA graph batch sizes:

Concurrency	Decoder graph batch sizes
8	1, 2, 4, 8
32	1, 2, 4, 8, 16, 24, 32
64	1, 2, 4, 8, 16, 32, 48, 64

PyTorch encoder CUDA graph buckets:

Concurrency	Encoder graph batch sizes	Token buckets	Sequence buckets
8	1, 2	128, 256, 512, 1,024	64, 128, 256, 512
32	1, 2, 4, 8	128, 256, 512, 1,024, 2,048, 4,096	64, 128, 256, 512
64	1, 2, 4	128, 256, 512, 1,024, 2,048, 4,096	64, 128, 256, 512

Mixed encoder-decoder CUDA graphs are enabled, with encoder-token buckets derived from the encoder runner's captured keys. At concurrency 64, the encoder batch-eight capture bucket is omitted because the full relative-attention layout set exceeds the 80 GB GPU during capture; encoder admission therefore uses batch-four microbatches.

The legacy TensorRT path reuses one BF16 encoder/decoder engine pair built with maximum batch size 64:

Engine	Max input	Max sequence	Max batched tokens	Opt batched tokens	KV cache
Encoder	512	512	16,384	8,192	Disabled
Decoder	1	129	8,192	64	Paged

Both legacy engines use BF16 BERT-attention, GPT-attention, and GEMM plugins with input-padding removal enabled. Context FMHA is disabled because the legacy T5 implementation does not support T5 relative attention bias through that path.

Encoder CUDA graph correctness validation

Before the relative-position correction, graph replay reused relative-position bias from capture instead of rebuilding it for replayed sequence lengths. For the same eight greedy requests, that produced 158 tokens, seven EOS stops, and one 128-token length stop, versus the eager reference's 32 tokens, eight EOS stops, and no length stops.

After the fix, the graph-enabled smoke run exactly matches eager execution: 32 generated tokens, eight EOS stops, and no length stops. All three full graph-enabled runs also have the expected 1,021 natural EOS stops and three length stops.

Validation

focused executor, warmup, model-engine, sampler, and LLM-args unit tests cover graph admission and replay, stream and TP synchronization, fixed sequence-slot staging, greedy completion, no-repeat-ngram fallback, and configuration validation
BART and T5 continuous-admission integration tests verify encoder graph replay and mixed decoder graph replay; both are assigned to H100 PyTorch post-merge L0
targeted BART/T5 CUDA-graph integration run: 2 passed
BART-large-CNN and FLAN-T5 Large PyTorch/legacy TensorRT benchmarks at concurrency 8, 32, and 64 on deterministic 1,024-request workloads
pre-commit formatting, lint, safety, test-list, and type-analysis hooks
DCO commit-message hooks

Dev Engineer Review

Adds asynchronous encoder scheduling with microbatch thresholds, deadlines, CUDA events, and tensor-parallel synchronization.
Adds encoder and mixed encoder-decoder CUDA graph capture and replay.
Adds persistent host-buffer input preparation and native bindings for encoder-decoder batches.
Adds fixed sequence-slot reuse and a fast single-step greedy sampler path.
Preserves the general sampler path for unsupported decoding features.
Adds and validates encoder_cuda_graph_config and enable_encoder_decoder_mixed_cuda_graph.
Updates documentation, supported-model guidance, API manifests, and configuration tests.
Main risks are asynchronous stream coordination, staging-buffer lifetime, mixed-graph key selection, encoder fallback behavior, and fast-path eligibility. Targeted tests cover these areas.

QA Engineer Review

Adds executor tests for encoder warmup, admission boundaries, fallback scheduling, asynchronous completion, tensor readiness, and tensor-parallel publication.
Adds model-engine tests for deferred two-pass warmup and fixed-sequence-slot staging and output restoration.
Adds sampler tests for no-repeat-ngram fallback and greedy EOS/length completion handling.
Adds TorchLlmArgs tests for encoder graph configuration validation and mixed-graph settings.
Adds BART and T5 continuous-admission tests for encoder and mixed decoder graph replay.
Updates CI test lists:
- l0_h100.yml: adds BART and T5 continuous-admission tests.
- l0_dgx_h100.yml: adds the two-GPU BART continuous-admission test.
- l0_l40s.yml: removes the superseded T5 mixed-context test.
The new integration tests are covered by H100 CI entries. Unit tests do not require test-list entries.
Verdict: sufficient.

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

cascade812 · 2026-08-03T21:45:12Z

/bot run

tensorrt-cicd · 2026-08-03T21:51:44Z

PR_Github #63565 [ run ] triggered by Bot. Commit: 15b2d67 Link to invocation

tensorrt-cicd · 2026-08-03T23:32:07Z

PR_Github #63565 [ run ] completed with state FAILURE. Commit: 15b2d67
/LLM/main/L0_MergeRequest_PR pipeline #51531 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

cascade812 · 2026-08-04T03:12:45Z

/bot run

tensorrt-cicd · 2026-08-04T03:18:54Z

PR_Github #63639 [ run ] triggered by Bot. Commit: 15b2d67 Link to invocation

tensorrt-cicd · 2026-08-04T04:08:28Z

PR_Github #63639 [ run ] completed with state FAILURE. Commit: 15b2d67
/LLM/main/L0_MergeRequest_PR pipeline #51595 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

cascade812 · 2026-08-04T19:18:39Z

/bot run

tensorrt-cicd · 2026-08-04T19:25:11Z

PR_Github #63824 [ run ] triggered by Bot. Commit: b7b15eb Link to invocation

tensorrt-cicd · 2026-08-04T21:44:45Z

PR_Github #63824 [ run ] completed with state SUCCESS. Commit: b7b15eb
/LLM/main/L0_MergeRequest_PR pipeline #51765 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

cascade812 · 2026-08-04T23:55:55Z

/bot run

tensorrt-cicd · 2026-08-05T00:02:40Z

PR_Github #63867 [ run ] triggered by Bot. Commit: 477ac44 Link to invocation

tensorrt-cicd · 2026-08-05T02:45:18Z

PR_Github #63867 [ run ] completed with state FAILURE. Commit: 477ac44
/LLM/main/L0_MergeRequest_PR pipeline #51809 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

cascade812 · 2026-08-05T06:38:15Z

/bot run

tensorrt-cicd · 2026-08-05T06:44:54Z

PR_Github #63967 [ run ] triggered by Bot. Commit: 477ac44 Link to invocation

pengbowang-nv

Attention part change LGTM

tensorrt-cicd · 2026-08-05T11:08:17Z

PR_Github #63967 [ run ] completed with state FAILURE. Commit: 477ac44
/LLM/main/L0_MergeRequest_PR pipeline #51903 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02

The greedy fast path doesn't screen out requests carrying penalties, so the penalty step is skipped for them. Disaggregated generation servers run almost entirely single-beam greedy decode, so they'd hit this on nearly every step.

mikeiovine

Mostly looks good, the penalty issue is the only blocker

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

…-perf Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com> # Conflicts: # tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py

cascade812 · 2026-08-05T21:47:52Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-05T21:53:50Z

PR_Github #64122 [ run ] triggered by Bot. Commit: 01ae2d2 Link to invocation

cascade812 added 3 commits July 21, 2026 14:33

[None][perf] accumulate encoder work during decode

1cc01f1

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

[None][perf] prepare mixed decoder batches natively

6fd8dc8

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

[None][perf] reuse stable greedy sampling metadata

433dcb9

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

github-actions Bot assigned cascade812 Jul 22, 2026

cascade812 added 6 commits July 26, 2026 10:17

[None][perf] optimize BART encoder-decoder execution

f3a8e10

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

[None][perf] enable encoder CUDA graphs for encoder-decoder models

ebccb5d

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

Merge origin/main into guiju/encoder-decoder-perf

0b6cc7c

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

[None][perf] improve encoder-decoder CUDA graph handling

04ca506

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

[None][fix] stabilize encoder-decoder CUDA graph capture

4f981ac

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

Merge origin/main into guiju/encoder-decoder-perf

0c32c18

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

cascade812 added the api-compatible Accepted LLM API contract change that is backwards-compatible label Jul 30, 2026 — with ChatGPT Codex Connector

cascade812 marked this pull request as ready for review July 30, 2026 18:39

cascade812 requested review from a team as code owners July 30, 2026 18:39

cascade812 requested review from PerkzZheng, QiJune, Shixiaowei02, allisonlim-nv, arysef, brnguyen2 and pcastonguay July 30, 2026 18:39

This was referenced Aug 4, 2026

[None][perf] Use FlashInfer MXFP8 GEMM for MiniMax-M3 decode #17237

Closed

[None][feat] Enforce multimodal encoder runtime budgets with budgeted output storage #16051

Open

fix failed tests

b7b15eb

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

arysef approved these changes Aug 4, 2026

View reviewed changes

address failed tests

477ac44

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

pengbowang-nv approved these changes Aug 5, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/attention_backend/trtllm.py Outdated

Shixiaowei02 requested a review from chienchunhung August 5, 2026 11:24

Shixiaowei02 requested changes Aug 5, 2026

View reviewed changes

mikeiovine reviewed Aug 5, 2026

View reviewed changes

cascade812 added 2 commits August 5, 2026 14:08

address comments

47b0116

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into guiju/encoder-decoder…

01ae2d2

…-perf Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com> # Conflicts: # tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py

cascade812 requested review from Shixiaowei02 and mikeiovine August 5, 2026 21:45

Uh oh!

Conversation

cascade812 commented Jul 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

User interface

Performance

BART-large-CNN

Detailed latency percentiles

Encoder input-length distribution

Configuration

FLAN-T5 Large

Detailed throughput and latency measurements

Output checks

Workload

Encoder input-length distribution

Reference output-length distribution

Configuration

Encoder CUDA graph correctness validation

Validation

Dev Engineer Review

QA Engineer Review

Uh oh!

cascade812 commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

cascade812 commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

cascade812 commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

cascade812 commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

cascade812 commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

pengbowang-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

Shixiaowei02 left a comment

Choose a reason for hiding this comment

Uh oh!

mikeiovine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cascade812 commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

cascade812 commented Jul 22, 2026 •

edited by coderabbitai Bot

Loading