Dynamic Routing for vLLM

[terrykong]

We need a router sidecar that queries the telemetry from vllm and determine which vLLM instance has the least load and send more prompts there.

[terrykong]

Example router from dynamo: https://github.com/ai-dynamo/dynamo/blob/main/examples/llm/components/kv_router.py

[dchichkov]

There's this repo - https://github.com/VectorInstitute/vector-inference

• This repository provides an easy-to-use solution to run inference servers on Slurm-managed computing clusters using vLLM.

Note also:
https://docs.vllm.ai/projects/production-stack/en/latest/user_manual/router/cmd.html

Note, slurm autoscale, with an option to backfill slurm capacity is the next obvious feature.

[euronymous-aithal]

@terrykong this is the updated standalone Router : https://github.com/ai-dynamo/dynamo/tree/main/examples/deployments/router_standalone

[terrykong]

thanks for the pointer @euronymous-aithal . converting this discussion to an issue.

#1210

please continue discussion on 1210