[None][feat] Disagg coordinator + multi-process orchestrator fleet by reasonsolo · Pull Request #15905 · NVIDIA/TensorRT-LLM

reasonsolo · 2026-07-03T04:51:25Z

TL;DR:

add num_workers: N, N>1 to disagg config yaml to enable multiprocess orchestrator fleet.
One possible drawback: /perf_metrics is still served by single orchestrator, if multi-processed, you may need to poll /perf_metrics for multiple times until it gives empty data.

DSv4 KVCaware router 6P1D c2832 experiment

    ┌──────────────────────────────_────────_────────_────────┐
  │            metric            │ fleet1 │ fleet4 │ fleet8 │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ req/s                        │ 59.3   │ 90.2   │ 98.0   │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ TTFT p50 (s)                 │ 39.979 │ 15.404 │ 14.892 │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ TTFT p95 (s)                 │ 60.879 │ 42.359 │ 37.675 │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ TTFT p99 (s)                 │ 67.652 │ 93.412 │ 60.745 │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ sys tok/s (server)           │ 20,040 │ 30,214 │ 32,580 │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ tok/s/GPU (server)           │ 626    │ 944    │ 1,018  │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ sys tok/s (client-est)       │ 14,603 │ 23,875 │ 25,193 │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ tok/s/GPU (client-est)       │ 456    │ 746    │ 787    │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ cache hit                    │ 97.0%  │ 95.2%  │ 95.2%  │
  ├──────────────────────────────┼────────┼────────┼────────┤
  │ output speed p50 (tok/s/req) │ 90.2   │ 49.4   │ 48.4   │
  └──────────────────────────────┴────────┴────────┴────────┘

Conv-router 6P1D C2832 experiment:

  ┌─────────────────────────────────────_──────────────────────_──────────────────────_──────────────────────┐
  │               Metric                │        fleet1        │        fleet4        │        fleet8        │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ Request rate (req/s)                │ 127.88               │ 149.47               │ 145.90               │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ TTFT p50 / p95 / p99 (s)            │ 7.23 / 12.63 / 15.49 │ 3.58 / 10.77 / 14.96 │ 3.84 / 11.82 / 16.26 │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ Sys throughput _ server (tok/s)     │ 41,818               │ 48,229               │ 47,594               │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ _ output tok/s/GPU (server)         │ 1,307                │ 1,507                │ 1,487                │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ Sys throughput _ client-est (tok/s) │ 28,342               │ 32,299               │ 32,184               │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ _ output tok/s/GPU (client-est)     │ 886                  │ 1,009                │ 1,006                │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ Cache hit                           │ 96.6%                │ 96.4%                │ 96.4%                │
  ├─────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
  │ Output speed p50 (tok/s/req)        │ 26.1                 │ 26.4                 │ 26.6                 │
  └─────────────────────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┘

Implement a coordinator/worker model:

disaggregated (num_workers >1) becomes a pure coordinator server that owns the ctx/gen routers, readiness, cluster/worker events, and serves only the internal /select, /finish, /cluster_info, /health API (coordinator_server.py).
The worker fleet runs over a shared listening socket (SO_REUSEPORT) on the public port, rebuilt from a stateless import-string factory (create_worker_app).
Placement is split on Router: extract_routing_key (client/worker side) + select_by_key / finish_by_handle (coordinator side). Round-robin -> empty key, conversation -> conversation_id (handle-based load release), centralized -> block hashes. The worker holds a CoordinatorHttpRouter that posts the key to the coordinator; single-process calls the router directly.
New DisaggCoordinator abstraction (disagg_coordinator.py): DisaggCoordinatorService (in-process, owns routers) and CoordinatorClient (worker, delegates over HTTP). OpenAIDisaggregatedService reads ctx_router/gen_router off the coordinator and drives get_next_server / finish_request uniformly, so serving is identical in both modes.
Remove router_http_server.py, disagg_app.py, RemoteHttpRouter and the remote_http router type, plus their tests; add test_coordinator_worker.py.

Verified in-container (gb200): test_coordinator_worker 2/2 across 5 repeats, test_per_rank_routing + test_centralized_kv_cache_router + test_openai_disagg_service 76 passed.

Summary by CodeRabbit

New Features
- Expanded disaggregated serving to support coordinator-managed worker fleets, including external coordinator setups and multi-worker deployments.
- Added improved request ID handling, routing coordination, and richer performance timing details for disaggregated requests.
Bug Fixes
- Improved health and readiness behavior for disaggregated serving.
- Added faster failure when a coordinator is unavailable.
Documentation
- Updated disaggregated serving docs with new topology examples, request ID format details, and configuration guidance.
Tests
- Added end-to-end and integration coverage for coordinator-driven routing and disaggregated request handling.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

reasonsolo · 2026-07-07T11:16:11Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-07T11:21:53Z

PR_Github #57997 [ run ] triggered by Bot. Commit: 2e56da7 Link to invocation

tensorrt-cicd · 2026-07-07T14:27:00Z

PR_Github #57997 [ run ] completed with state SUCCESS. Commit: 2e56da7
/LLM/main/L0_MergeRequest_PR pipeline #46667 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

pcastonguay · 2026-07-07T16:26:12Z

Any idea why the KV cache hit rate drops when using KV aware routing with fleet > 1?

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

reasonsolo · 2026-07-15T09:21:11Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-15T09:26:56Z

PR_Github #59424 [ run ] triggered by Bot. Commit: 1278df7 Link to invocation

tensorrt-cicd · 2026-07-15T09:30:58Z

PR_Github #59397 [ run ] completed with state ABORTED. Commit: 1dc6833

Link to invocation

nv-xtf

LGTM

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

tensorrt-cicd · 2026-07-15T13:30:38Z

PR_Github #59424 [ run ] completed with state SUCCESS. Commit: 1278df7
/LLM/main/L0_MergeRequest_PR pipeline #47893 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

reasonsolo · 2026-07-16T04:55:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-16T05:01:19Z

PR_Github #59612 [ run ] triggered by Bot. Commit: 9b2a8cd Link to invocation

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

reasonsolo · 2026-07-16T07:34:04Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-16T07:40:17Z

PR_Github #59659 [ run ] triggered by Bot. Commit: 647370c Link to invocation

tensorrt-cicd · 2026-07-16T07:44:35Z

PR_Github #59612 [ run ] completed with state ABORTED. Commit: 9b2a8cd

Link to invocation

tensorrt-cicd · 2026-07-16T11:01:49Z

PR_Github #59659 [ run ] completed with state SUCCESS. Commit: 647370c
/LLM/main/L0_MergeRequest_PR pipeline #48094 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

reasonsolo · 2026-07-17T07:32:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-17T07:37:37Z

PR_Github #59926 [ run ] triggered by Bot. Commit: 2922df3 Link to invocation

tensorrt-cicd · 2026-07-17T11:06:39Z

PR_Github #59926 [ run ] completed with state SUCCESS. Commit: 2922df3
/LLM/main/L0_MergeRequest_PR pipeline #48328 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

reasonsolo · 2026-07-17T12:22:46Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-17T12:29:14Z

PR_Github #59978 [ run ] triggered by Bot. Commit: 2922df3 Link to invocation

tensorrt-cicd · 2026-07-17T16:32:18Z

PR_Github #59978 [ run ] completed with state SUCCESS. Commit: 2922df3
/LLM/main/L0_MergeRequest_PR pipeline #48372 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

github-actions Bot assigned reasonsolo Jul 3, 2026

reasonsolo force-pushed the feat/deepseek_v4_coordinator_disagg branch 5 times, most recently from dfe13a6 to 492d8ce Compare July 7, 2026 05:46

reasonsolo marked this pull request as ready for review July 7, 2026 05:51

reasonsolo requested review from a team as code owners July 7, 2026 05:51

reasonsolo requested review from Tabrizian, arysef, bo-nv, hchings and nv-guomingz and removed request for a team July 7, 2026 05:51

reasonsolo changed the title ~~[None][feat] Disagg coordinator + orchestrator fleet~~ [None][feat] Disagg coordinator + multi-process orchestrator fleet Jul 7, 2026

reasonsolo requested a review from a team as a code owner July 7, 2026 10:13

reasonsolo force-pushed the feat/deepseek_v4_coordinator_disagg branch from f938c4e to c3cc4e3 Compare July 8, 2026 09:33

reasonsolo requested review from a team as code owners July 8, 2026 09:33

[None][fix] synchronize delegated router hashing

1278df7

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

nv-xtf approved these changes Jul 15, 2026

View reviewed changes

Shixiaowei02 approved these changes Jul 15, 2026

View reviewed changes

Comment thread tensorrt_llm/llmapi/disagg_utils.py

Comment thread tensorrt_llm/serve/router.py

[None][fix] address coordinator reservation feedback

a780efd

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

QiJune reviewed Jul 15, 2026

View reviewed changes

Comment thread tensorrt_llm/serve/disagg_coordinator.py Outdated

QiJune reviewed Jul 15, 2026

View reviewed changes

Comment thread tensorrt_llm/serve/disagg_coordinator.py Outdated

QiJune reviewed Jul 15, 2026

View reviewed changes

Comment thread tensorrt_llm/commands/serve.py

chienchunhung approved these changes Jul 16, 2026

View reviewed changes

Comment thread tensorrt_llm/serve/router.py

Comment thread tensorrt_llm/serve/router.py Outdated

reasonsolo added 3 commits July 15, 2026 20:02

[None][fix] address fleet coordinator review feedback

0186c59

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

[None][fix] move fleet state sync off request path

a882b17

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

[None][test] start coordinator client before readiness check

9b2a8cd

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

[None][fix] harden coordinator routing hot path

647370c

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

reasonsolo added 2 commits July 17, 2026 14:34

Merge branch 'main' into feat/deepseek_v4_coordinator_disagg

2aa0577

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

[None][fix] update KV router integration test metadata usage

2922df3

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

Uh oh!

Conversation

reasonsolo commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

reasonsolo commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

pcastonguay commented Jul 7, 2026

Uh oh!

reasonsolo commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

nv-xtf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

Uh oh!

Uh oh!

reasonsolo commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

reasonsolo commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

reasonsolo commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

reasonsolo commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

reasonsolo commented Jul 3, 2026 •

edited

Loading