Dropped sentences in ASR transcript w/ Parakeet v3/v2

[hamzaq2000]

Transcribed this video using Parakeet v2 and v3.

# Clone repo
git clone https://github.com/FluidInference/FluidAudio.git
cd FluidAudio

# Download and convert to 16 KHz 16-bit wav
uvx "yt-dlp[default]" -f bestaudio GT_sXIUJPUo -o "temp_GT_sXIUJPUo.%(ext)s"
ffmpeg -i "temp_GT_sXIUJPUo."* -acodec pcm_s16le -ac 1 -ar 16000 cowen.wav
rm "temp_GT_sXIUJPUo."*

# Build
swift build -c release

# Transcribe
swift run fluidaudio transcribe cowen.wav --model-version v2
swift run fluidaudio transcribe cowen.wav --model-version v3

I also transcribed the same file using parakeet-mlx to compare.

All results can be found here: cowen_results.zip

Inspected the first ~1.5 pages of all transcripts and noticed:

• Fluid Audio parakeet v3 drops a lot of sentences. See fluid_audio_parakeet_v3/cowen_fluidaudio_parakeet_v3_errors.pdf in the zip file.
• Fluid Audio parakeet v2 still drops sentences but much fewer (only 1 vs 8 with v3). See fluid_audio_parakeet_v2/cowen_fluidaudio_parakeet_v2_errors.pdf in the zip file.
• parakeet-mlx drops no sentences at all. See parakeet_mlx_parakeet_v3/cowen_parakeet_mlx_parakeet_v3 in the zip file.

Wondering if this is a known issue? Or is it just this one audio file causing a malfunction somehow?

Unfortunate, because the FluidAudio parakeet-tdt implementations are otherwise the fastest on Apple Silicon, faster than parakeet-mlx. Hopefully fixable; I will try to understand the model architecture and code and see if I'm able to find the cause as well.

[BrandonWeng]

Thank you for the detailed results, that helps a ton. We did have notice issues here in the past, but when we compared it with the Nemo model, it was also missing a lot of text

#99 (comment)

I suspect its how we're handling the blank tokens and the Decoder state between chunks. I found that it actually works much better if we dont carry the decoder state between different chunks from experiemnting with VAD based chunking here:

main...vad-asrr

My intuition tells me its fixable, but haven't had time to dig deeper into it

[hamzaq2000]

Wow interesting that the nemo implementation also drops sentences; perhaps parakeet-mlx should be the reference implementation then 😄

I see. Hope to dig into the model architecture; will be a good learning opportunity for me.

[BrandonWeng]

Wow interesting that the nemo implementation also drops sentences; perhaps parakeet-mlx should be the reference implementation then 😄

I see. Hope to dig into the model architecture; will be a good learning opportunity for me.

This was surprising for us as well, but its worth noting that this was the streaming NeMo API. Batched might perform better

[hamzaq2000]

Ah, gotcha.

[BrandonWeng]

Interesting, there are similar reports online. Thanks for sharing your repo with me. Resetting the LSTM state for each chunk with some overlap and running LCS merging seems to help, but it degrades the state. I wonder if that’s related to non-English languages not working. I recall we also did stateless decoding for v2, and that wasn’t working when we migrated to v3. One of the major changes was to respect the LSTM state and pass it along during decoding.

• Sherpa-ONNX: Users observed dropped words with parakeet-tdt-0.6b-v3-int8. The root cause was the decoder allowing zero-duration blanks and unlimited per-frame token emissions, advancing the timeJump past valid frames.
  → Fix: [PR #2606](Fix TDT decoding for NeMo TDT transducers k2-fsa/sherpa-onnx#2606) clamps skips and caps tokens per frame, addressing the missing “doing” word noted in [issue #2605](Missing words with nemo-parakeet-tdt-0.6b-v3-int8 k2-fsa/sherpa-onnx#2605).

• NeMo: [Issue #14430](Long deletions when reduce the right context using streaming_infer NVIDIA-NeMo/NeMo#14430) documents “long deletions” when streaming parakeet-tdt-0.6b-v2 with reduced right context. NVIDIA confirmed it’s due to the model’s reliance on sustained decoder state—shorter context exaggerates negative timeJump drift, causing whole phrases to vanish.

• MLX port: Reports a related failure — a 15 s timestamp jump in the second sentence wipes out text despite high token confidence ([senstella/parakeet-mlx#43](Abnormal Timestamps Cause Sentence Loss - 15-Second Time Jump in Second Sentence senstella/parakeet-mlx#43)). This points to corrupted carry-over state rather than acoustic error.

• Language-model fusion: When NeMo’s MALSD + LM fusion is enabled, Parakeet v2/v3 sometimes emit empty or truncated hypotheses even though raw decoding is fine ([NeMo #14907](Accuracy issue with MALSD decoding with LM fusion NVIDIA-NeMo/NeMo#14907)). Maintainers link this to interactions between stored decoder state and external LM search.

• Active tuning: NVIDIA continues adjusting Parakeet’s timestamp/head logic — see [PR #14912](Fix TDT beam search timestamp alignment NVIDIA-NeMo/NeMo#14912) (fixes beam-search timestamp offsets by persisting token durations) and related streaming PRs — highlighting that consistent LSTM state + duration bookkeeping remains a moving target for v3 models.

[BrandonWeng]

Finally got some more time to look into this. Note that the parakeet-mlx version isn't great either, but there's def problem with our implementation. Was able to bring down the error rate in this PR #177

Comparing with the results here: https://openai-whisper.hf.space/?__theme=dark&deep_link=OYajcTavFaU

Although Whisper text may not be as accurate but it generally doesn't skip words as badly as the parakeet-tdt-v3 models

Current main

FA v3 vs MLX v3
Metrics
 WER               9.25%
 CER               8.12%
 Reference Words   12252
 Hypothesis Words  12766

FA v3 vs MLX v2
Metrics
 WER               15.34%
 CER               12.50%
 Reference Words   12252
 Hypothesis Words  13186

FA v3 vs Whisper-v3-large

Metrics
 WER               15.34%
 CER               12.50%
 Reference Words   12252
 Hypothesis Words  13186

MLX v2 vs Whisper-v3-large
Metrics
 WER               7.99%
 CER               5.71%
 Reference Words   13186
 Hypothesis Words  12766

MLX v3 vs Whisper-v3-large
Metrics
 WER               7.99%
 CER               5.71%
 Reference Words   13186
 Hypothesis Words  12766


whisper-v3-large vs 11labs

Metrics
 WER               9.67%
 CER               6.68%
 Reference Words   12993
 Hypothesis Words  13186

New

FA v3 vs MLX v3
Metrics
 WER               4.01%
 CER               3.01%
 Reference Words   12766
 Hypothesis Words  12666

FA v2 vs MLX v2
Metrics
 WER               4.69%
 CER               2.87%
 Reference Words   12766
 Hypothesis Words  12616

FA v2 vs Whisper-v3-large

Metrics
 WER               7.65%
 CER               5.64%
 Reference Words   13186
 Hypothesis Words  12616

FA v2 vs Whisper-v3-large
Metrics
 WER               9.39%
 CER               7.04%
 Reference Words   13186
 Hypothesis Words  12666