Skip to content

vllm.md

Latest commit

 

History

History
75 lines (57 loc) · 15.9 KB

vllm.md

File metadata and controls

75 lines (57 loc) · 15.9 KB
Raw

Running the model on vLLM

English | 繁體中文 · Back to the README

The model can also run on a vLLM server: one the page starts for you in Docker (below), or any server, on this machine or on a Linux machine with a GPU, while the page stays on yours.

A server of your own

For a server of your own, set VLLM_URL to where it listens:

# Where vLLM runs (vLLM 0.32 or newer, on a Linux machine with a GPU). --revision is the commit in REVISIONS, so the
# server and the page read the same tokenizer and chat template. The other flags let the page see more (below).
vllm serve Qwen/Qwen3.5-0.8B --revision 2fc06364715b967f1860aea9cf38778875588b17 --enable-scale-out
# Or Gemma 4 E2B with its MTP drafter, saying which tokens were guesses:
vllm serve google/gemma-4-E2B-it --revision 3e22461f65e89153144f8adb70e3b8c2cc9845a7 --enable-scale-out \
  --speculative-config '{"method": "mtp", "model": "google/gemma-4-E2B-it-assistant", "revision": "2d874ef7d29f9a30599a1e4b3c1cbc9595f005df", "num_speculative_tokens": 6}' \
  --per-request-spec-decode-metrics detailed
# Or Qwen3.5 with the MTP layer in its own checkpoint, which transformers cannot use:
vllm serve Qwen/Qwen3.5-0.8B --revision 2fc06364715b967f1860aea9cf38778875588b17 --enable-scale-out \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' --per-request-spec-decode-metrics detailed
# Or with sampling masks, which show how many candidates were left in place of the model's own probabilities (below):
vllm serve Qwen/Qwen3.5-0.8B --revision 2fc06364715b967f1860aea9cf38778875588b17 --enable-scale-out \
  --return-sampling-mask --logprobs-mode processed_logprobs

# Where the page runs (without just: VLLM_URL=http://localhost:8000 uv run streamlit run streamlit_app.py):
VLLM_URL=http://localhost:8000 just run

The Model menu then lists what the server serves. A model of a family the page does not support yet is marked ⚠, which the page tells from its config.json alone, and a new tab starts on the first model it does support. Picking a marked model says why it does not run and offers a GitHub issue asking for it, with the title and description already filled in: the model and its model_type, never the server's address. If the server was started with --api-key, give the page the same key in VLLM_API_KEY (the variable vllm serve reads it from). If the server cannot hold the model's whole context in its KV cache, lower it with --max-model-len; the page's context limit follows the server's.

What the page sees

vllm_backend.py is a second Backend (backend.py) next to simple_backend.py; the harness, the tools and the page do not change. The prompt stays exact: the page downloads the model's config, tokenizer and chat template into models/ (not the weights), renders and tokenizes the prompt itself, and sends the server the token ids, so the server reads exactly what the Context tab shows. Each token comes back as its id with its log probability and those of the most likely tokens at its position. With vLLM's default --logprobs-mode raw_logprobs those are the model's own distribution, as in simple_backend.py, and give the popover's model probability and top 5 candidates. How much more the page sees depends on the server's flags; at load, the page asks the server, with one-token requests, which of them it was started with:

Server flags Where the steps go What the page shows
none /v1/completions the probability, the top 5, the rank only when the token is one of them; text only
--enable-scale-out /inference/v1/generate also every token's rank, and the images, sound and video the server takes
--return-sampling-mask --logprobs-mode processed_logprobs too the same the candidates left and the sampling probability, in place of the model probability, the heat and the top 5: vLLM sends sampling masks only with processed log probabilities, and then refuses temperature 0 and top-k 0, so the sidebar starts at 0.05 and 1
--per-request-spec-decode-metrics summary / detailed either MTP's counts in each step; with detailed, the underlines on the guesses too, once the step is done

What the page leaves out it does not make up (backend.TOKEN_DETAILS). In detail:

  • --logprobs-mode decides what the numbers are, and no route says which one the server runs, so at load the page asks for one token at temperature 1 with top-k 2 and the 5 most likely tokens: in a processed mode the 3 that top-k cut come back as -9999, and log probabilities, unlike logits, add up to at most 1. With processed_logprobs (which --return-sampling-mask needs; vLLM refuses to start without it, or without its Model Runner V2, which it picks by default where it can), a token's log probability is its probability in the distribution it was drawn from, after temperature, top-k and top-p, so the popover shows it as the sampling probability, with the rank, and shows no model probability, heat or top 5. How many candidates were left comes from the sampling mask, or, without one, shows when top-k / top-p cut some of the most likely tokens listed. With raw_logits or processed_logits, the numbers are not probabilities, and the page shows none of them as one. Which of the top 5 top-k / top-p cut (gray in simple_backend.py's popover) shows in no mode outside greedy decoding: the model's own probabilities and what the sampler did never come back together. The Model tab says which mode the server runs.
  • The rank counts the tokens more likely than the chosen one, as in simple_backend.py, so a token tied for first (common with bfloat16 logits) is #1. /inference/v1/generate's rank counts the tokens tied with it as well (vLLM 0.32 compares with >=), and with speculative decoding it compares the logits, while the log probabilities it lists can tie where the logits do not. So while a listed token is no likelier than the chosen one, every likelier one is listed too, and the page counts the rank from the log probabilities it shows; a token below all of them keeps the server's rank, which may still count ties the server did not list.
  • Greedy decoding (temperature 0) needs no server to say that one candidate was left and drawn with probability 1, so the popover shows that on any server that takes it. A server started with --return-sampling-mask does not: vLLM refuses temperature 0 and top-k 0 there, on every route, so the sidebar's sliders stop at 0.05 and 1: a recommended temperature below 0.05 moves up to 0.05, and an unlimited top-k (0) to 128, the slider's top. Without --enable-scale-out no response shows the masks, so at load the page also asks such a server (one with processed log probabilities) for a greedy token, and a refusal says the same.
  • Images, sound and video go next to the placeholders the chat template wrote, as the server's content_parts: an image as PNG, turned upright by its EXIF; sound as WAV, decoded here; a video as the file itself, whose frames the server picks. The server expands each placeholder into the file's tokens. So that the Context tab shows the prompt as the server reads it, the page asks it for that expansion first, with a request that generates one token; if the server expands differently when generating, the step is an error. At load the page sends the server a tiny image, clip and video to see which kinds it takes; a kind it refuses (or every kind, without --enable-scale-out) cannot be attached, the line above the input box says so, and the Model tab lists why.
  • MTP is the server's (--speculative-config), so the sidebar's switch is off and disabled. With --per-request-spec-decode-metrics summary (which vLLM takes only with --speculative-config), each step says how many guesses were kept and how many decoding forward passes ran. vLLM counts a verify step's kept guesses before it cuts the step at a stop token, so when the stop token is one of them, the guesses after it count too. With detailed it also says how many guesses each verify step kept, from which the page marks the guesses: the first token comes from the prefill, then each verify step gives the guesses it kept and one token of the model's own. vLLM does not count a step without guesses, so the page marks the tokens only when the counts account for every one of them, and only once the step is done, counting as kept only the guesses up to the stop token, as simple_backend.py does; what a turned-down guess was, the server does not say. Both models decode speculatively on a vLLM server with a GPU, and every real-server test passes with it: on a DGX Spark's GB10, Qwen3.5 0.8B went from about 113 to 164 tok/s with 2 guesses, and Gemma 4 E2B from about 41 to 108 tok/s with 4. vLLM 0.32's CPU image runs neither: Qwen3.5's first request takes the engine down in a Triton kernel the CPU build lacks (the one that keeps its Gated DeltaNet state in step with the guesses), and Gemma 4's drafter sets up CUDA graphs as it loads.
  • Prefill is the time to the first token, the network included. vLLM keeps the KV cache of what a prompt shares with the last one (prefix caching), so from step 2 on it reads only the new part, where simple_backend.py reads the whole prompt every step. With media, the expansion request before it has already read most of the prompt. The page times prefill and decoding as the tokens reach it, so Streamlit's full garbage collection after every script run is off (.streamlit/config.toml): it held the GIL for up to a quarter of a second, as a question started and on every redraw, and the step's tokens then reached the page all at once, at thousands of tok/s.
  • The Model tab names the server and its version; the specs come from config.json and the context limit from the server. The device, the precision and the parameter count are the server's and are not shown, and the folder card lists the weights as on the server only.
  • Not identical to generate(): vLLM batches requests and has kernels of its own, so greedy output need not match transformers' token for token, as simple_backend.py's does.

How it works tells the two backends apart in tabs at every step where they differ, from what each reads in models/ to MTP, and has a section on the Backend interface they share, all with their code.

vLLM in Docker, started by the page

Without VLLM_URL, the sidebar's What runs the model switches between transformers and vLLM (Docker). Picking vLLM lets go of the model loaded here first (release_model() clears the cache and hands the memory back, on CUDA cuBLAS's workspace too, which would otherwise keep the whole model's memory; a switch made while the model loads takes effect once it has loaded, since Streamlit's fast reruns are off), then agent/model_backends/vllm_docker.py downloads the model into models/ if it is not there yet and starts the server in docker-compose/ with docker compose up -d, showing what Docker and vLLM print meanwhile. Picking transformers again runs docker compose down, and so does quitting the app (Ctrl+C), so only one of the two holds the model. What runs the model is one choice for every tab, as the model in memory is: a tab open on transformers follows another that picks vLLM, and back, and a new tab or a reload joins the server with the model it runs, rather than loading the model here beside it. Switching models on vLLM goes through docker compose up again, which replaces the container; a rerun with the server already up starts nothing. A start that failed (no Docker, Docker not running, out of memory) says why in the page's language and is tried again only when you press Try again; the switch back to transformers stays in the sidebar.

# The same server by hand (VLLM_MODEL and VLLM_REVISION default to Qwen3.5 0.8B at its pinned commit); the weights must be in models/ first (just download Qwen/Qwen3.5-0.8B, or the model in VLLM_MODEL):
just vllm-up                              # on the CPU, any machine
just vllm-up gpu                          # on NVIDIA GPUs (Linux, NVIDIA Container Toolkit)
VLLM_URL=http://localhost:8000 just run
just vllm-down                            # stops the server

Without just, just vllm-up is docker compose -f docker-compose/compose.yaml up -d, just vllm-up gpu adds -f docker-compose/compose.gpu.yaml after the first file, and just vllm-down is docker compose -p agent-loop-vllm down.

  • Where it runs. Where Docker has the nvidia runtime, compose.gpu.yaml goes on top and vLLM's own CUDA image (vllm/vllm-openai:v0.32.0) runs on the GPUs; elsewhere, a Mac included, compose.yaml runs vLLM's CPU image inside Docker's Linux VM. AGENT_LOOP_VLLM_DEVICE=cpu or gpu decides instead, and VLLM_PORT moves the port (8000). On the GPU the KV cache is 2 GiB, as on the CPU (--kv-cache-memory-bytes, VLLM_KV_CACHE_MEMORY_BYTES), rather than 92% of the GPU's memory, and vLLM wants only 30% of that memory free to start (--gpu-memory-utilization, VLLM_GPU_MEMORY_UTILIZATION): at its default 0.92 it refuses on a small card beside the page's own CUDA context, and on a GPU that shares the machine's memory, such as a DGX Spark's.
  • MTP. On a GPU the server decodes speculatively by default: Gemma 4 with its drafter, Qwen3.5 with the MTP layer in its own checkpoint (model_families.VLLM_SPECULATIVE, handed to Compose as VLLM_SPECULATIVE_CONFIG), and each step underlines the guesses it kept. The sidebar's MTP switch turns it off or back on, which starts the server again the other way, and every tab follows. vLLM 0.32's CPU image can decode neither model speculatively, so on a CPU the switch is disabled and says why. Started by hand, the GPU file runs without it.
  • What the server reads. It never downloads: models/ is mounted read-only with HF_HUB_OFFLINE=1, so it loads exactly the commit in REVISIONS, as the page does. It listens on 127.0.0.1 only, like the page. It is started with --enable-scale-out, so the page sees every token's rank and the server takes images, sound and video, and keeps vLLM's default raw_logprobs, so the popover shows the model's own probabilities and the top 5. vLLM's compile cache is kept in a Docker volume, so only the first start of a model compiles.
  • The CPU image (docker-compose/vllm-cpu/) is vllm/vllm-openai-cpu:v0.32.0 with two fixes (patch_vllm.py): vLLM's own #61992, which came a day after 0.32.0 and without which every model's first request fails on a CPU, and one of ours, which makes the input of a CPU linear layer contiguous, without which the first clip Gemma 4 hears takes the engine down. On the CPU, compose.yaml sets the KV cache to 2 GiB (VLLM_CPU_KVCACHE_SPACE; otherwise vLLM wants 92% of Docker's memory), skips measuring the image encoder at startup (over 10 minutes on a CPU), and lets one step take up to 30 minutes rather than vLLM's 5 before it gives up on the engine.
  • On a DGX Spark (NVIDIA GB10, whose GPU shares the machine's 128 GB): on the GPU, Qwen3.5 0.8B starts in about 2 minutes and decodes about 113 tok/s, Gemma 4 E2B starts in about 3 minutes and decodes about 41 tok/s, and the real-server tests pass for both, every kind of media Gemma reads included. At vLLM's default gpu_memory_utilization (0.92) the server refuses to start there: vLLM takes the machine's available memory for the GPU's free memory, about 108 of 122 GiB. On the same machine's CPU, Qwen3.5 0.8B starts in about 1.3 minutes and decodes about 23 tok/s.
  • On a Mac (Apple M5, 10 cores given to Docker): Qwen3.5 0.8B starts in about 1.5 minutes, takes about 9 GB of Docker's memory and decodes about 10 tok/s. Gemma 4 E2B needs more memory than Docker Desktop's default: with 16 GB it is killed while loading (the page says so and how to give Docker more), with 24 GB it starts in 3 to 5 minutes and decodes about 4.6 tok/s, but checking at load which media the server takes runs its image, audio and video encoders on the CPU and takes about 10 minutes the first time.