Summary
When an LLM emits a text preamble before a tool_call in the same assistant turn (Hermes / Qwen chat templates — common with vLLM, sglang, and any model fine-tuned for the Hermes function-calling format), inference.llm.LLMStream._parse_choice silently drops every tool_call streaming delta until finish_reason arrives. During the ~0.8–1.2s the model spends serializing the tool_call JSON arguments, nothing reaches the downstream consumer.
Streaming TTS plugins (Cartesia, ElevenLabs streaming) keep the preamble sentence buffered, waiting for either a FlushSentinel (#3933) or end-of-stream — which only happens after the tool_call args finish. Net result: ~590–1000 ms of dead air before the agent speaks the preamble on every tool turn. Measured end-to-end against live Cartesia.
Why #3933 doesn't fully solve it
PR #3933 added the FlushSentinel primitive (great — we use it). The official example (examples/voice_agents/flush_llm_node.py) flushes a hardcoded "One moment…" string and yields the sentinel after consuming the whole LLM stream. That works if you're willing to replace the model's preamble with a scripted one.
But if you want to preserve the model's own preamble (which is the whole point of letting the model generate it), there is no signal exposed by the OpenAI inference plugin that a custom llm_node can hook into to know "preamble is done, tool_call mode started." The tool deltas that would carry that signal are silently dropped in _parse_choice.
Root cause
livekit-agents/livekit/agents/inference/llm.py, LLMStream._parse_choice (lines 439–542 on v1.5.7 / current main):
- Tool deltas (
delta.tool_calls) accumulate into self._tool_call_id, _fnc_name, _fnc_raw_arguments, _tool_extra, _tool_index. No ChatChunk is yielded.
- A
ChatChunk is only emitted when finish_reason in ("tool_calls", "stop") (line 487) or when a new tool index appears mid-stream (line 455).
- The explicit
if not delta.content and not delta_extra: return None at the bottom drops every intermediate tool-only delta.
So a llm_node consuming the stream sees: preamble text chunks → silence (during entire tool-args serialization) → final tool_calls ChatChunk at finish. The TTS, which only flushes on FlushSentinel or stream end, sits on the preamble.
Repro
- Provider: any OpenAI-compatible endpoint serving a Hermes/Qwen-style model (reproducible with Qwen2.5 / Qwen3 MoE on vLLM or sglang, and any Hermes-2 fine-tune).
- System prompt that encourages "say one sentence, then call tool X".
- Streaming TTS (Cartesia / ElevenLabs streaming) on
tts_node.
- Measure delta between LLM text-content done and first TTS audio frame.
Consistent ~600–1000 ms gap on every tool turn.
Proof / measurements
End-to-end against a Qwen MoE on sglang + live Cartesia, through production code paths:
Tool+preamble turns: avg (first_audio − end_input) = -587 ms
(negative = audio starts BEFORE LLM stream ends)
Sample timeline (with a flush emitted at first tool_call delta):
1941 ms FlushSentinel yielded
2123 ms first audio frame (182 ms after flush)
2562 ms tool_call_final ChatChunk arrives
2563 ms end_input (LLM stream ends ~440 ms AFTER audio started)
Without that flush, first audio is gated on end_input, so audio starts ~440 ms after the LLM finishes — the dead-air gap.
Proposed shapes — open to direction
Three plausible shapes, in decreasing order of how invasive they are:
A. Auto-emit FlushSentinel from inference.llm.LLMStream on first tool.function.name. Most idiomatic post-#3933 — the sentinel becomes a first-class element of the inference stream. Changes the stream's element type, and _metrics_monitor_task will need guards similar to PR #4787 (langgraph adapter).
B. Emit a marker ChatChunk with delta.extra["tool_start"] = True. Type-stable — stream elements remain ChatChunk. Consumers (custom llm_node, voice/generation.py) convert to FlushSentinel themselves. Less invasive but pushes the conversion to every consumer that wants the benefit.
C. Callback hook on LLMStream (on_tool_start: Callable[[], None] | None = None). Most conservative but adds API surface for something that arguably belongs in the stream itself.
Leaning (A) if you're comfortable with the _event_ch type widening (precedent in #4787); otherwise (B).
Related issues / PRs
Happy to send a PR once we agree on shape. If the preferred path is "users should write their own llm_node + sentinel and the inference plugin stays as-is," guidance on how a custom llm_node is supposed to know preamble ended without a signal from the stream would be appreciated.
Thanks for #3933 — the primitive made this solvable; just want to make the win available without per-team workarounds.
cc @longcw @theomonnom
Summary
When an LLM emits a text preamble before a tool_call in the same assistant turn (Hermes / Qwen chat templates — common with vLLM, sglang, and any model fine-tuned for the Hermes function-calling format),
inference.llm.LLMStream._parse_choicesilently drops every tool_call streaming delta untilfinish_reasonarrives. During the ~0.8–1.2s the model spends serializing the tool_call JSON arguments, nothing reaches the downstream consumer.Streaming TTS plugins (Cartesia, ElevenLabs streaming) keep the preamble sentence buffered, waiting for either a
FlushSentinel(#3933) or end-of-stream — which only happens after the tool_call args finish. Net result: ~590–1000 ms of dead air before the agent speaks the preamble on every tool turn. Measured end-to-end against live Cartesia.Why #3933 doesn't fully solve it
PR #3933 added the
FlushSentinelprimitive (great — we use it). The official example (examples/voice_agents/flush_llm_node.py) flushes a hardcoded "One moment…" string and yields the sentinel after consuming the whole LLM stream. That works if you're willing to replace the model's preamble with a scripted one.But if you want to preserve the model's own preamble (which is the whole point of letting the model generate it), there is no signal exposed by the OpenAI inference plugin that a custom
llm_nodecan hook into to know "preamble is done, tool_call mode started." The tool deltas that would carry that signal are silently dropped in_parse_choice.Root cause
livekit-agents/livekit/agents/inference/llm.py,LLMStream._parse_choice(lines 439–542 on v1.5.7 / current main):delta.tool_calls) accumulate intoself._tool_call_id,_fnc_name,_fnc_raw_arguments,_tool_extra,_tool_index. No ChatChunk is yielded.ChatChunkis only emitted whenfinish_reason in ("tool_calls", "stop")(line 487) or when a new tool index appears mid-stream (line 455).if not delta.content and not delta_extra: return Noneat the bottom drops every intermediate tool-only delta.So a
llm_nodeconsuming the stream sees: preamble text chunks → silence (during entire tool-args serialization) → final tool_calls ChatChunk at finish. The TTS, which only flushes onFlushSentinelor stream end, sits on the preamble.Repro
tts_node.Consistent ~600–1000 ms gap on every tool turn.
Proof / measurements
End-to-end against a Qwen MoE on sglang + live Cartesia, through production code paths:
Without that flush, first audio is gated on
end_input, so audio starts ~440 ms after the LLM finishes — the dead-air gap.Proposed shapes — open to direction
Three plausible shapes, in decreasing order of how invasive they are:
A. Auto-emit
FlushSentinelfrominference.llm.LLMStreamon firsttool.function.name. Most idiomatic post-#3933 — the sentinel becomes a first-class element of the inference stream. Changes the stream's element type, and_metrics_monitor_taskwill need guards similar to PR #4787 (langgraph adapter).B. Emit a marker
ChatChunkwithdelta.extra["tool_start"] = True. Type-stable — stream elements remainChatChunk. Consumers (customllm_node,voice/generation.py) convert toFlushSentinelthemselves. Less invasive but pushes the conversion to every consumer that wants the benefit.C. Callback hook on
LLMStream(on_tool_start: Callable[[], None] | None = None). Most conservative but adds API surface for something that arguably belongs in the stream itself.Leaning (A) if you're comfortable with the
_event_chtype widening (precedent in #4787); otherwise (B).Related issues / PRs
FlushSentinel. Sets up the mechanism this fix would use.llm_noderesult? #2417, BufferedTokenStream never yields the completed tokens (sentence or word) till the next token arrives causing delays in TTS Speech generation #3798, How to force speech synthesis for partial llm_node result? #3529 (closed): all variants of "TTS waits on LLM stream end", closed pointing at add flush for llm_node #3933. None proposed auto-flushing at the tool boundary from the inference plugin side.FlushSentinelforwarding for the LangGraph adapter; relevant precedent for_metrics_monitor_tasknon-ChatChunkhandling.Happy to send a PR once we agree on shape. If the preferred path is "users should write their own llm_node + sentinel and the inference plugin stays as-is," guidance on how a custom
llm_nodeis supposed to know preamble ended without a signal from the stream would be appreciated.Thanks for #3933 — the primitive made this solvable; just want to make the win available without per-team workarounds.
cc @longcw @theomonnom