[TRTLLM-13409][fix] hard-kill all ranks when one rank's executor loop crashes by JunyiXu-nv · Pull Request #16592 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-07-20T05:22:30Z

Dev Engineer Review

Implemented multi-rank “rank crash hard-kill” escalation with a configurable grace period (TLLM_RANK_CRASH_HARD_KILL_GRACE), where invalid/unset values default to 10s and negative values disable world hard-kill.
Ensured escalation logic is best-effort and non-throwing (does not raise if kill propagation fails/cancellation occurs), preserving the original executor exception for debugging.
Added a cancellable RankCrashKillWatchdog and integrated it into PyExecutor._event_loop_wrapper so the watchdog is armed before executor cleanup and the hard-kill is attempted afterward, preserving the original fire deadline.
Added unit coverage for single-rank exemption, grace timing/delay, disabling/invalid env handling, non-raising behavior, watchdog cancel behavior, deadline preservation, and crash vs clean-exit wiring.
No configuration or test-list files were changed.

QA Engineer Review

Test changes (unit)

tests/unittest/_torch/executor/test_hang_detector_kill.py
- Added/extended coverage for:
  - test_rank_crash_kill_single_rank_is_noop
  - test_rank_crash_kill_fires_for_multi_rank
  - test_rank_crash_kill_sleeps_grace_before_kill
  - test_rank_crash_kill_disabled_by_negative_grace
  - test_rank_crash_kill_invalid_grace_uses_default
  - test_rank_crash_kill_never_raises
  - test_watchdog_kills_while_caller_blocks
  - test_watchdog_not_armed_for_single_rank
  - test_watchdog_not_armed_when_disabled
  - test_watchdog_cancel_prevents_the_kill
  - test_kill_keeps_original_deadline_on_handover
  - test_event_loop_wrapper_kills_world_on_crash
  - test_event_loop_wrapper_kills_world_when_cleanup_raises
  - test_event_loop_wrapper_kills_world_when_watchdog_cannot_arm
  - test_event_loop_wrapper_no_kill_on_clean_exit
  - test_event_loop_wrapper_no_kill_when_loop_raises_after_shutdown
  - test_event_loop_wrapper_no_kill_when_enclosing_context_manager_raises
- CI list coverage:
  - Included in tests/integration/test_lists/test-db/l0_sanity_check.yml (module unittest/_torch/executor/test_hang_detector_kill.py)
tests/unittest/executor/test_proxy_fast_death.py
- Modified:
  - test_pool_session_shutdown_never_blocks_after_release
- CI list coverage:
  - Included in tests/integration/test_lists/test-db/l0_a10.yml (module unittest/executor/test_proxy_fast_death.py)

Verdict

Sufficient (relevant unit test modules are already covered in CI test-db via the test list entries above).

Description

When a rank's executor loop dies on an exception, the rank stops participating in collectives but nothing tells its peers: every peer blocks in its next collective until its own HangDetector fires 300s later, and the whole multi-GPU test session burns that long for an error that was already known (the A4 AutoDeploy catches are this signature: peers crash, the survivor wedges in ADP until the 300s backstop).

Single-rank worlds are exempt (no peers to unblock), and the kill helper never raises (it runs in a finally where an exception would mask the loop's original error).

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

JunyiXu-nv · 2026-07-20T05:23:46Z

/bot run --disable-fail-fast

coderabbitai · 2026-07-20T05:25:01Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Adds configurable rank-crash hard-kill handling with a grace period, optional watchdog, and defensive error handling. Executor crash cleanup now starts the watchdog before cleanup and invokes direct world termination afterward, with tests covering policy and ordering.

Changes

Rank crash handling

Layer / File(s)	Summary
Configurable rank-crash kill policy `tensorrt_llm/_torch/pyexecutor/hang_detector.py`	Adds environment-based grace parsing, optional disabling, world-level hard-kill behavior, and daemon watchdog startup.
Executor crash integration `tensorrt_llm/_torch/pyexecutor/py_executor.py`	Tracks event-loop crashes, starts the watchdog before cleanup, and invokes direct hard-kill handling after cleanup.
Kill behavior validation `tests/unittest/_torch/executor/test_hang_detector_kill.py`, `tests/unittest/executor/test_proxy_fast_death.py`	Tests grace handling, watchdog applicability and cancellation, kill error suppression, crash cleanup ordering, exception preservation, clean exits, and shutdown state setup.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PyExecutor_event_loop_wrapper
  participant RankCrashKillWatchdog
  participant executor_loop_cleanup
  participant hard_kill_on_rank_crash
  PyExecutor_event_loop_wrapper->>RankCrashKillWatchdog: Arm after event-loop crash
  PyExecutor_event_loop_wrapper->>executor_loop_cleanup: Execute local cleanup
  RankCrashKillWatchdog->>hard_kill_on_rank_crash: Trigger after grace period
  PyExecutor_event_loop_wrapper->>hard_kill_on_rank_crash: Invoke after cleanup with original deadline

Suggested reviewers: schetlur-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main change and follows the required [ticket][type] summary format.
Description check	✅ Passed	The description explains the problem and solution, and the checklist is checked, but Test Coverage is left blank.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-20T05:29:40Z

PR_Github #60270 [ run ] triggered by Bot. Commit: cf7fdc1 Link to invocation

tensorrt-cicd · 2026-07-20T08:49:03Z

PR_Github #60270 [ run ] completed with state SUCCESS. Commit: cf7fdc1
/LLM/main/L0_MergeRequest_PR pipeline #48629 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-07-21T02:48:21Z

/bot run

tensorrt-cicd · 2026-07-21T02:55:20Z

PR_Github #60547 [ run ] triggered by Bot. Commit: cf7fdc1 Link to invocation

tensorrt-cicd · 2026-07-21T03:08:02Z

PR_Github #60547 [ run ] completed with state FAILURE. Commit: cf7fdc1
/LLM/main/L0_MergeRequest_PR pipeline #48859 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-07-21T05:04:02Z

/bot run

tensorrt-cicd · 2026-07-21T05:09:51Z

PR_Github #60580 [ run ] triggered by Bot. Commit: cf7fdc1 Link to invocation

nv-xtf

LGTM — One non-blocking question inline about kill reachability if cleanup blocks on a PP send handle.

tensorrt-cicd · 2026-07-21T09:19:36Z

PR_Github #60580 [ run ] completed with state FAILURE. Commit: cf7fdc1
/LLM/main/L0_MergeRequest_PR pipeline #48890 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-07-30T10:13:52Z

/bot run

tensorrt-cicd · 2026-07-30T10:19:38Z

PR_Github #62761 [ run ] triggered by Bot. Commit: dd070b7 Link to invocation

tensorrt-cicd · 2026-07-30T12:52:48Z

PR_Github #62761 [ run ] completed with state SUCCESS. Commit: dd070b7
/LLM/main/L0_MergeRequest_PR pipeline #50891 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-07-31T03:27:51Z

Post-rebase run #50891 — zero test failures

The rebase did its job. Across 37 stages that uploaded results, there are no failing testcases at all. test_row_linear — the stale-base failure that killed the two previous runs — does not appear as a failure anywhere.

What failed instead is stage-level, not test-level:

[DGX_H100-PyTorch-3-attempt-2]  "Stage Failed" — "Stage run failed without result"

That is the synthetic testcase generateStageFailTestResultXml emits when a stage dies without producing a results*.xml. Note it is attempt-2, i.e. the stage had already been retried once. Thirteen further tests across six B200/B300 stages are recorded unfinished, which is the signature of a cancellation cascade rather than six independent problems.

This is the failure class I measured earlier at roughly 1 pipeline in 6: the pipeline goes red while no test does, so the verdict carries no information about the change.

Re-running.

JunyiXu-nv · 2026-07-31T03:27:52Z

/bot run

tensorrt-cicd · 2026-07-31T03:34:06Z

PR_Github #62939 [ run ] triggered by Bot. Commit: dd070b7 Link to invocation

tensorrt-cicd · 2026-07-31T08:11:51Z

PR_Github #62939 [ run ] completed with state FAILURE. Commit: dd070b7
/LLM/main/L0_MergeRequest_PR pipeline #51056 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-01T09:34:15Z

/bot run

JunyiXu-nv · 2026-08-03T01:59:18Z

/bot run

… crashes When a rank's executor loop dies on an exception, the rank stops participating in collectives but nothing tells its peers: every peer blocks in its next collective until its own HangDetector fires 300s later, and the whole multi-GPU test session burns that long for an error that was already known (the A4 AutoDeploy catches are this signature: peers crash, the survivor wedges in ADP until the 300s backstop). Escalate at error time instead: after the loop's local cleanup has woken rank-local waiters, hard-kill the world via the ST-1 propagation path. A grace period (TLLM_RANK_CRASH_HARD_KILL_GRACE, default 10s, negative disables) lets the cleaner error paths win the race first -- the stashed error reaches rank-local response waiters, the init-phase ready handshake returns the real exception to the proxy, and the worker future completes with the original error -- so the client reports the actual failure rather than a bare worker death. Single-rank worlds are exempt (no peers to unblock), and the kill helper never raises (it runs in a finally where an exception would mask the loop's original error). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…blocks or raises The kill sat after _executor_loop_cleanup() in the finally block, so it was skippable in exactly the situations it exists for: cleanup blocking without bound (unbounded wait() on a PP send handle wedged by the crash) or cleanup raising (aborting the finally before the kill). Either way peers fall back to burning 300s in their own HangDetectors. Arm a daemon watchdog thread BEFORE cleanup that fires the kill at crash + grace regardless of cleanup progress, and nest the post-cleanup kill in its own finally so a raising cleanup cannot skip it. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…pool-session shutdown test The test builds MpiPoolSession via __new__ (a real spawn is neither needed nor wanted), but MpiPoolSession.shutdown() now reads n_workers (added by NVIDIA#16456 while the test was in flight) and _wait_shutdown, both set only in __init__. Provide them explicitly and drop the waive. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…rand peers The kill was armed for ANY exception escaping the executor loop wrapper, including one raised on the way out of an already-shutting-down loop. A hardware A/B on 2x GB200 (mpirun -n 2 trtllm-llmapi-launch trtllm-bench --tp 2) showed it turning a benign late exception -- raised after event_loop() returned, i.e. after the shutdown request was processed and all work was done -- into a whole-job SIGKILL: mpirun exit 137 at crash+10s, where the same injection on the merge-base logged the error, let the worker thread die, and completed the benchmark with exit 0. It fired during the memory-profiling dry run, before the benchmark even started: PyExecutor is constructed and shut down twice per process and that first shutdown is a normal lifecycle event. Three changes: - crashed = not self.is_shutdown. Once is_shutdown is set every rank has processed the shutdown broadcast, so no peer is waiting on this one and there is nothing to escalate. - Scope the flag to event_loop() itself via an inner try. Teardown of the enclosing host-profiler / GC context managers is not a stranded-peer condition either. - Make the watchdog cancellable and give it an explicit deadline. It was a bare daemon thread that slept the grace and killed with no cancel path, so once armed it fired at crash+10s even if the process went on to shut down cleanly and would have exited 0. It also double-armed: the watchdog and the post-cleanup hard_kill_on_rank_crash each ran their own timer. The watchdog now covers only the window where cleanup may block forever; once cleanup returns the caller cancels it and carries the kill inline on the watchdog's ORIGINAL deadline, so exactly one timer is live at any moment and the handover cannot push the kill out by a second grace. A genuine mid-loop crash still arms the watchdog and still hard-kills the world -- the behavior the kill exists for is unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…bed kill If cancel() ever regresses, the watchdog thread outlives monkeypatch teardown and SIGKILLs the pytest process instead of failing the test. Join it inside the test, while propagate_hard_kill is still stubbed. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…is_shutdown The previous commit keyed the decision off `not self.is_shutdown`, on the premise that is_shutdown means "every rank has processed the shutdown broadcast". That premise is false, and the mistake silently disabled the kill for this feature's most common trigger. `_handle_errors` sets `self.is_shutdown = True` rank-locally whenever an error is fatal, and `classify_error` returns `immediate_fatal` for a CUDA illegal address / device-side assert / launch failure, bypassing the error budget entirely. The only "broadcast" that follows is `enqueue_shutdown_request()`, which pushes into this process's own queue -- peers are told nothing. `should_stop_processing` additionally requires empty active/waiting queues, so the loop keeps entering collectives after the flag is set. So: rank 3 of 4 takes cudaErrorIllegalAddress in the forward pass, _handle_errors sets is_shutdown and returns None, and the next unguarded statement (guided_decoder.execute(batch_outputs['logits'])) raises TypeError on None. The wrapper computed crashed=False, armed nothing, and ranks 0-2 blocked in their next NCCL collective for the full 300s -- exactly the failure this PR exists to remove. Gate on an explicit completion sentinel instead. `_event_loop_completed` is set at the three executor loops' normal-exit `break` sites and nowhere else, so it answers the actual question: did event_loop() reach its own termination? A raise after that point (profiler/hang-detector __exit__, or the enclosing context managers) is a teardown error; anything else -- now including a failure before the loop ever started, which the previous inner try narrowed away -- strands peers and escalates. Also corrects two overstated claims from the previous commit. The watchdog/post-cleanup "double-arm" never produced an extra or later kill: earliest-wins gave crash+grace before, and max(cleanup_end, crash+grace) = crash+grace after. cancel() cannot spare a rank that exits cleanly either -- it is followed one line later by the same kill on the same deadline. The handover machinery is kept because one timer is clearer than two, but all the protection against a spurious SIGKILL rests on the predicate above. Log fixes: the disarm message drops from ERROR to DEBUG (it printed "cancelled before the grace elapsed" moments before the world was SIGKILLed, which reads during triage as "the kill was called off"), and the countdown now logs the remaining time rather than the full grace on the handover path. Tests: a rank-local-fatal regression test that fails against the previous predicate; a pre-loop-failure test; coverage for the already-elapsed deadline (the max(0.0, ...) clamp is the only thing keeping a negative sleep from raising into the blanket except and dropping the kill); and an AST check that every `break` in the three loops is preceded by the sentinel, so a new normal-exit path cannot make clean shutdowns look like crashes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

… process-wide sleep patch monkeypatching time.sleep is process-wide, so any background thread that sleeps during these two tests appends its own entry into the asserted list. Assert the ordering these tests are about instead of exact list equality. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…le on the deadline clamp Five guard-quality defects from review. None changed runtime behavior on a healthy run, but two recreated the conditions under which the predicate has already regressed twice. 1. The AST guard enforced the wrong invariant. It required EVERY own `break` in the three loops to be preceded by the sentinel, but only a break exiting the outer `while True` terminates the event loop, and _executor_loop / _executor_loop_overlap already contain inner `for` loops. Worse, its failure message told the reader "every normal exit must set it", so a contributor who added an inner-loop break would have been instructed to set the sentinel while the loop was still running -- making every later rank-local crash look like a clean shutdown and silently disabling the kill, the same class as the previous blocker. Now: exactly one loop-terminating break per loop, located by recursing through if/try/with but stopping at nested for/while/def. Mutation-tested: inner `for _z in (): break` without the sentinel PASSES; a missing sentinel on the outer break FAILS; a second outer break FAILS. 2. The load-bearing reset was untested. Deleting `self._event_loop_completed = False` from _event_loop_wrapper left the whole suite green while a second loop run on the same executor misread a genuine crash as a clean shutdown (verified: run-2 events go from [watchdog, cleanup, cancel, kill] to [cleanup]). The one "initialization" test only grepped __init__ -- the site that is NOT load-bearing -- which invited deleting the real one as redundant. Added a behavioral two-invocation test plus a source assertion on the wrapper. 3. The rationale on the max(0.0, ...) deadline clamp was false. The comment and docstring claimed it was the only thing preventing a negative sleep from raising into the blanket except and dropping the kill. It is not: _wait_out_kill_grace guards with `remaining > 0`, and Event.wait() returns immediately for a negative timeout (probe: _wait_out_kill_grace(-100, None) -> True, zero sleep calls). A false "this guard protects X" comment is how the previous regressions got through, and someone trusting it could drop the real guard as redundant. Corrected both, moved the load-bearing note onto the `> 0` guard, and added a test asserting sleep is never called with a negative argument. 4. Hardening the two grace-ordering tests against the process-wide time.sleep patch weakened them: membership-only assertions accepted sleeping the grace TWICE before killing -- exactly the crash + 2*grace bug this series introduced with its two independent timers. Pinned the counts to 1 while still tolerating an unrelated thread's sleep. The handover test now pins time.monotonic and asserts the exact remaining duration instead of racing a 0.3s real sleep against a 0.55s bound on a loaded CI node. 5. The AST guard checked neither the assigned value nor the target object: `self._event_loop_completed = False` before the break passed, and so did `self.dist._event_loop_completed = True`. A value inversion turns every clean shutdown into a whole-job SIGKILL and was invisible. Now requires Constant True assigned to an attribute of Name('self'); both mutants fail. Also documents why _executor_loop_pp sets the sentinel before its Stage-5 drain: reaching there means every rank observed should_stop_processing, so the drain consumes only a rank-local queue and a raise in it strands nobody. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…p, not walk order _executor_loop_pp contains four while loops, one of them (the Stage-5 drain) a sibling of the event loop under the same with-block. Selecting by ast.walk order happened to pick the right one, but a body reorder would silently repoint the guard at the drain. Select by the `True` test and assert uniqueness instead. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

main and this branch each introduced an alias for the same module -- main used `hang_detector_module`, this branch used `hd_module`. The rebase resolved the import-block conflict in favour of main's name, but later commits in this series reintroduced `hd_module.` call sites, leaving the tip importing one alias and calling the other (19 undefined references, NameError on collection). Converge on main's `hang_detector_module` so the branch follows trunk convention rather than the other way round. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-08-03T02:16:36Z

/bot run

tensorrt-cicd · 2026-08-03T02:23:14Z

PR_Github #63332 [ run ] triggered by Bot. Commit: 7093b33 Link to invocation

tensorrt-cicd · 2026-08-03T04:19:01Z

PR_Github #63332 [ run ] completed with state FAILURE. Commit: 7093b33
/LLM/main/L0_MergeRequest_PR pipeline #51327 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-03T08:32:49Z

/bot run --extra-stage "DGX_B200-4_GPUs-PyTorch-3, DGX_H100-4_GPUs-PyTorch-DeepSeek-1"

tensorrt-cicd · 2026-08-03T08:38:29Z

PR_Github #63398 [ run ] triggered by Bot. Commit: 7093b33 Link to invocation

tensorrt-cicd · 2026-08-03T09:21:39Z

PR_Github #63398 [ run ] completed with state FAILURE. Commit: 7093b33
/LLM/main/L0_MergeRequest_PR pipeline #51377 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-04T03:07:38Z

/bot run

tensorrt-cicd · 2026-08-04T03:13:51Z

PR_Github #63637 [ run ] triggered by Bot. Commit: 7093b33 Link to invocation

tensorrt-cicd · 2026-08-04T03:59:04Z

PR_Github #63637 [ run ] completed with state FAILURE. Commit: 7093b33
/LLM/main/L0_MergeRequest_PR pipeline #51591 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv requested review from a team as code owners July 20, 2026 05:22

JunyiXu-nv requested review from YihuiLu512, nv-xtf, reasonsolo and zhaoyangwang-nvidia July 20, 2026 05:22

github-actions Bot assigned JunyiXu-nv Jul 20, 2026

JunyiXu-nv force-pushed the dev-junyix-feat-rank-crash-hard-kill branch from 03e5e9e to cf7fdc1 Compare July 20, 2026 05:23

nv-xtf approved these changes Jul 21, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Outdated

zhaoyangwang-nvidia requested changes Jul 21, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Outdated

JunyiXu-nv force-pushed the dev-junyix-feat-rank-crash-hard-kill branch from cf7fdc1 to ec5a475 Compare July 22, 2026 03:53

JunyiXu-nv requested review from a team as code owners July 22, 2026 03:53

JunyiXu-nv requested review from brnguyen2, schetlur-nv, xinhe-nv, yiqingy0 and yuanjingx87 July 22, 2026 03:53

JunyiXu-nv force-pushed the dev-junyix-feat-rank-crash-hard-kill branch from ec5a475 to c0c1a43 Compare July 22, 2026 07:34

JunyiXu-nv added 10 commits August 3, 2026 02:14

JunyiXu-nv force-pushed the dev-junyix-feat-rank-crash-hard-kill branch from dd070b7 to 7093b33 Compare August 3, 2026 02:16

brnguyen2 mentioned this pull request Aug 3, 2026

[TRTLLM-14903][fix] Free partially-allocated warmup dummy KV blocks and count spec extra tokens in warmup block estimates #17162

Open

2 tasks

Uh oh!

Conversation

JunyiXu-nv commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Test changes (unit)

Verdict

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

JunyiXu-nv commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

JunyiXu-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

JunyiXu-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

nv-xtf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

Uh oh!

JunyiXu-nv commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

JunyiXu-nv commented Jul 31, 2026

Post-rebase run #50891 — zero test failures

Uh oh!

JunyiXu-nv commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

JunyiXu-nv commented Aug 1, 2026

Uh oh!

JunyiXu-nv commented Aug 3, 2026

Uh oh!

JunyiXu-nv commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

JunyiXu-nv commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

JunyiXu-nv commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

JunyiXu-nv commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading