[Bug]: Speculative decoding + TorchSampler: unseeded requests share one RNG stream → identical/degenerate samples at low concurrency

[jiasenwu]

System Info

• TensorRT-LLM: 1.3.0rc25+deepl.mr.240.2.v26.6 (downstream build of 1.3.0rc25)
• TensorRT: 10.15.1.29
• torch: 2.13.0a0+8145d630e8.nv26.6.54250401
• Sampler: TorchSampler (default; TRTLLMSampler is deprecated)
• Serving: trtllm-serve, OpenAI-compatible endpoint
• Model: Qwen/Qwen3-8B
• Draft model: thoughtworks/Qwen3-8B-Eagle3
• GPU: NVIDIA H100 80GB HBM3
• Driver: 580.159.04 · CUDA: 13.3 (nvcc V13.3.33) · Python: 3.12.3

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Serve Qwen3-8B with speculative decoding (max_batch_size > 1):

trtllm-serve serve Qwen/Qwen3-8B --config sd.yaml

sd.yaml:

speculative_config:
  decoding_type: Eagle3
  max_draft_len: 4
  speculative_model: thoughtworks/Qwen3-8B-Eagle3
  eagle3_one_model: true
  eagle3_model_arch: llama3
  use_rejection_sampling: true

2. Fire independent, one-at-a-time (serial / low-concurrency) requests with temperature sampling and no seed:

from openai import OpenAI

c = OpenAI(base_url="http://<host>:<port>/v1", api_key="EMPTY")
model = c.models.list().data[0].id
prompt = "Once upon a time, there was a"

def sample(seed=None):
    kw = dict(model=model, prompt=prompt, max_tokens=12,
              temperature=0.7, top_p=0.95, extra_body={"top_k": 0})
    if seed is not None:
        kw["seed"] = seed
    return c.completions.create(**kw).choices[0].text

no_seed = [sample() for _ in range(200)]            # no seed
seeded  = [sample(seed=i) for i in range(200)]      # distinct seed each
print("no seed :", len(set(no_seed)), "/ 200 distinct")
print("seeded  :", len(set(seeded)),  "/ 200 distinct")

Repeat with speculative decoding disabled to compare.

Expected behavior

Two separate unseeded requests with identical prompts sample independently and — at temperature 0.7 — almost always differ. Without SD this holds: 192 / 200 distinct continuations.

actual behavior

With speculative decoding on, unseeded serial requests collapse to (near-)identical outputs:

condition (200 serial requests, same prompt)	distinct outputs
SD off, no seed	192 / 200 (healthy)
SD on, no seed	1–2 / 200 — effectively all identical (e.g. 200× " boy who loved to read. He wou…")
SD on, distinct seed per request	192–196 / 200 (healthy)

Single-token sampling (max_tokens=1, SD on, no seed) similarly collapses to a couple of distinct tokens instead of the true ~30–40-token distribution.

use_rejection_sampling does not affect this: the unseeded collapse is present both with use_rejection_sampling: true (1–2 / 200 distinct) and without it, and the per-request seed fixes it in both cases.

additional notes

When it appears / disappears

• Disappears at high concurrency: with many requests in flight simultaneously, sampling looks correct — so the collapse is easy to miss in throughput benchmarks and mainly bites low-QPS / single-stream traffic.
• Disappears with SD off (same model, same build).
• Fixed by a distinct per-request seed.
• n > 1 in a single request also collapses (the n samples are not independent), so best-of-N via n is affected regardless of concurrency.

Impact

• Silent loss of sampling diversity for unseeded requests under speculative decoding at low/moderate concurrency — repeated prompts, retries, and best-of-N return identical or near-identical outputs.
• Output depends on batch composition / slot assignment.
• Easy to miss because high-concurrency benchmarks look fine.

Workarounds

1. Send a distinct SamplingParams.seed per request (random or per-request-id-derived) — fully restores diversity at any concurrency in our tests.
2. For multiple samples of one prompt, issue N separate seeded requests instead of n > 1.
3. Or disable speculative decoding (loses the SD speedup).

Possibly related: the TorchSampler docstring in seed_manager.py warns that the pinned flashinfer-python reads only seed[0]/offset[0] per sampling call and separates rows by blockIdx.x; flashinfer PR #2345 ("add per-request generator support for sampling kernels") is tracking a fix. This may be the same underlying limitation.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[mikeiovine]

Should be fixed by #19505.