Environment: @livekit/agents 1.4.5 · @livekit/agents-plugin-elevenlabs 1.4.5 · @livekit/agents-plugin-deepgram 1.4.5 · @livekit/rtc-node 0.13.29 · Node v24.16.0 · Windows 11 (issue 1's contamination variant also reproduced on a LiveKit Cloud-hosted worker).
Pipeline: Silero VAD + Deepgram nova-3 (streaming) + external LLM + ElevenLabs TTS (eleven_multilingual_v2) + EnglishModel turn detection + interruption: { minDuration: 800, minWords: 2, mode: 'adaptive' }.
Component: agents-js voice/audio_recognition.ts + voice/agent_activity.ts (same design in the Python SDK).
Summary. When onEndOfTurn skips a user turn via the minWords gate ("skipping user input, word count below minimum interruption threshold"), the dropped final remains in AudioRecognition.audioTranscript. audioTranscript is cleared only on a committed turn or clearUserTurn() — there is no time-based expiry. Dropped sub-minWords finals therefore accumulate indefinitely.
Two user-visible failure modes (both device-reproduced):
- Backchannels kill agent speech. With
minWords: 2, isolated backchannels ("okay", "right") are individually dropped, but 2–3 of them across one uncommitted stretch accumulate to ≥2 words. The accumulated transcript then (a) passes the minWords word gate inside interruptByAudioActivity on the next VAD/detector trigger (pause→kill), or (b) commits as a normal multi-word turn at EOU, and userTurnCompleted interrupts the actively-playing reply. Observed: user backchanneled "Right." three times at natural ~2–3s cadence during a long reply → transcript "Right. Right. Right." committed → the playing reply was killed mid-sentence.
- Next-turn contamination. A dropped crumb prefixes the next genuinely-committed turn: user says "Okay." (dropped) then later "Stop." → the LLM receives "Okay. Stop." Observed on both a local worker and a LiveKit Cloud-hosted worker.
Expected. Backchannels the word gate deliberately rejects should not be able to re-enter the interruption/commit paths later by accumulation, and should not contaminate unrelated future turns.
Repro. Any voice agent with minWords: 2: play a long reply; say "okay" … (2s) … "right" … (2s) … "right" during it. The reply is interrupted by the accumulated commit; alternatively inspect audioTranscript after a single dropped "okay" — it persists until the next committed turn.
Suggested fix shape. Timestamp the retained transcript when onEndOfTurn returns false, and expire it on staleness (no new final and no active VAD speech for ~2–2.5s). A genuine hesitant interruption ("hey … can you stop") lands its next final well inside that window; a backchannel followed by silence never does. We run this as a runtime shim (staleness snapshot + not-speaking + no-pending-EOU-task guards), plus an immediate clear when the dropped crumb consists entirely of ack/filler vocabulary (rolling backchannels at natural 2–3s cadence re-arm any staleness window and otherwise accumulate past minWords into a reply-killing commit). An SDK-level fix could more simply exclude gate-dropped finals from the accumulator entirely (or key retention on content, not on commit status).
Environment:
@livekit/agents1.4.5 ·@livekit/agents-plugin-elevenlabs1.4.5 ·@livekit/agents-plugin-deepgram1.4.5 ·@livekit/rtc-node0.13.29 · Node v24.16.0 · Windows 11 (issue 1's contamination variant also reproduced on a LiveKit Cloud-hosted worker).Pipeline: Silero VAD + Deepgram nova-3 (streaming) + external LLM + ElevenLabs TTS (
eleven_multilingual_v2) +EnglishModelturn detection +interruption: { minDuration: 800, minWords: 2, mode: 'adaptive' }.Component:
agents-jsvoice/audio_recognition.ts+voice/agent_activity.ts(same design in the Python SDK).Summary. When
onEndOfTurnskips a user turn via theminWordsgate ("skipping user input, word count below minimum interruption threshold"), the dropped final remains inAudioRecognition.audioTranscript.audioTranscriptis cleared only on a committed turn orclearUserTurn()— there is no time-based expiry. Dropped sub-minWordsfinals therefore accumulate indefinitely.Two user-visible failure modes (both device-reproduced):
minWords: 2, isolated backchannels ("okay", "right") are individually dropped, but 2–3 of them across one uncommitted stretch accumulate to ≥2 words. The accumulated transcript then (a) passes theminWordsword gate insideinterruptByAudioActivityon the next VAD/detector trigger (pause→kill), or (b) commits as a normal multi-word turn at EOU, anduserTurnCompletedinterrupts the actively-playing reply. Observed: user backchanneled "Right." three times at natural ~2–3s cadence during a long reply → transcript "Right. Right. Right." committed → the playing reply was killed mid-sentence.Expected. Backchannels the word gate deliberately rejects should not be able to re-enter the interruption/commit paths later by accumulation, and should not contaminate unrelated future turns.
Repro. Any voice agent with
minWords: 2: play a long reply; say "okay" … (2s) … "right" … (2s) … "right" during it. The reply is interrupted by the accumulated commit; alternatively inspectaudioTranscriptafter a single dropped "okay" — it persists until the next committed turn.Suggested fix shape. Timestamp the retained transcript when
onEndOfTurnreturns false, and expire it on staleness (no new final and no active VAD speech for ~2–2.5s). A genuine hesitant interruption ("hey … can you stop") lands its next final well inside that window; a backchannel followed by silence never does. We run this as a runtime shim (staleness snapshot + not-speaking + no-pending-EOU-task guards), plus an immediate clear when the dropped crumb consists entirely of ack/filler vocabulary (rolling backchannels at natural 2–3s cadence re-arm any staleness window and otherwise accumulate pastminWordsinto a reply-killing commit). An SDK-level fix could more simply exclude gate-dropped finals from the accumulator entirely (or key retention on content, not on commit status).