Skip to content

feat(openengine): add sibling server for TRT-LLM#16336

Draft
connorcarpenter15 wants to merge 4 commits into
NVIDIA:mainfrom
connorcarpenter15:feat/openengine-server
Draft

feat(openengine): add sibling server for TRT-LLM#16336
connorcarpenter15 wants to merge 4 commits into
NVIDIA:mainfrom
connorcarpenter15:feat/openengine-server

Conversation

@connorcarpenter15

Copy link
Copy Markdown

Summary

  • add an optional OpenEngine gRPC sibling server to trtllm-serve while preserving the existing HTTP and native gRPC interfaces
  • share one LLM instance across protocols with protocol-neutral admission, cancellation, draining, request tracking, and exactly-once shutdown
  • implement generation, discovery, health, load, abort, drain, LoRA lifecycle, KV connector discovery, and KV event/statistics streaming
  • support aggregate and context-first prefill/decode roles, ordered image/video/audio media, per-request media options, guided decoding, logprobs, priority, cache controls, and lazy LoRA selection
  • fan out TRT-LLM's single-consumer KV and statistics queues so HTTP and OpenEngine do not compete

Why

This provides an engine-neutral process boundary for Dynamo without replacing TRT-LLM's current in-process, HTTP, or native gRPC integrations. The same TRT-LLM engine can serve HTTP and an OpenEngine sidecar endpoint in one process.

Protocol dependency

Validation

  • TensorRT-LLM wheel built successfully in the TRT-LLM development container
  • OpenEngine unit and fake-servicer suite: 64 passed
  • targeted context-first PEFT/LoRA prefill-decode test passed
  • native gRPC compatibility, conversion, cancellation, drain, fanout, LoRA, and shutdown coverage passed
  • computelab frontend smokes passed for Qwen3 aggregate text/LoRA, Qwen3 1P1D text/LoRA, Qwen3-VL image/video aggregate and 1P1D, and Phi-4 aggregate audio

Compatibility

OpenEngine is disabled unless --openengine-port is supplied. Existing HTTP behavior remains enabled, and the dedicated native --grpc mode is rejected only when combined with the new sibling server.

Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant