perf: cache whisper context between requests + releaseModel()#3
Open
andreayres wants to merge 2 commits into
Open
perf: cache whisper context between requests + releaseModel()#3andreayres wants to merge 2 commits into
andreayres wants to merge 2 commits into
Conversation
whisper_init_from_file() was called on every transcribe request and the context freed at the end, so each request paid a full model load from disk - seconds on mobile, dwarfing inference for short audio chunks and making near-real-time streaming transcription impossible with anything above the tiny model. Error paths before whisper_free() also leaked the context. Cache the context in a process-global keyed by model path, guarded by a mutex (whisper_full mutates the context, and Dart callers issue requests from short-lived isolates). A new "releaseModel" request type frees the cached context so callers can reclaim memory when done transcribing. Applied identically to src/ (Android), ios/Classes/ and macos/Classes/, which are copies of the same file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Companion to the native context cache: Whisper.releaseModel() sends the new "releaseModel" request so apps can drop the cached model from memory once transcription is finished. ReleaseModelRequest is a plain WhisperRequestDto implementation (no freezed codegen required), and _request() gains an ensureModel flag so releasing never triggers a model download. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
Author
Review pingLooking for a review on this perf fix — highest impact of the two related PRs. Why it mattersOn-device streaming STT was reloading the ggml model from disk on every short buffer flush. Measured path: ~9.7s audio → ~19.5s wall (~2× real-time), mostly load not inference. Caching the native whisper context (mutex + path key) should make 2nd+ buffers much closer to real-time. Companion app PR
After mergeApp still pins Happy to adjust API / naming if maintainers prefer a different release surface than |
Author
|
cc @beastoin @xuegao-tzx @guyluz11 — would love a maintainer eye on this when you have bandwidth (esp. the native cache + mutex approach). |
5 tasks
mdmohsin7
added a commit
to BasedHardware/omi
that referenced
this pull request
Jul 11, 2026
… teardown (#9438) ## Summary `CompositeTranscriptionSocket.stop()` / `disconnect()` tore down child sockets while `_status` was still `connected`, so each child's `onClosed` re-entered `_onSocketClosed` as an **unexpected** close: - Misleading log: `Primary socket closed (code: null)` - Double-disconnect of both children - False `onClosed` to `CaptureProvider` → reconnect snackbar + keep-alive churn on every intentional teardown (including every BLE reconnect) ## Fix Set `_status = disconnected` **before** stopping/disconnecting children so `_onSocketClosed`'s guard treats those callbacks as expected. ## Context Seen with custom STT + on-device Whisper composite mode: device BLE flaps cause intentional composite teardown; the spurious close path made each flap look like a transcription outage and triggered unnecessary reconnect UI. Related plugin work (context cache so reconnects are cheaper): BasedHardware/whisper_flutter_new#3 ## Test plan - [ ] Enable composite / on-device Whisper STT - [ ] Start recording, stop recording → no `Primary socket closed (code: null)` warning for intentional stop - [ ] Simulate BLE disconnect/reconnect while recording (or toggle device) → no false reconnect snackbar solely from composite teardown race - [ ] Intentional transcription settings change still reconnects cleanly <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/BasedHardware/omi/pull/9438?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img](https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a){kind=link}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
whisper_init_from_file()+whisper_free()on everytranscriberequest.src/main.cpp), iOS, and macOS native copies.whisper_free().Whisper.releaseModel()so apps can free the cached context on demand (plainReleaseModelRequest, no freezed codegen).Why
On-device streaming STT (e.g. Omi composite + on-device Whisper) flushes short buffers every few seconds. Reloading ggml-small from disk each time dominated wall time — e.g. 9.7s audio in ~19.5s (~2× real-time) was mostly model load, not inference. With a warm cache, small models can keep up with near-real-time use.
Notes for reviewers
whisper_fullmutates the context; the mutex serializes concurrent isolate requests.releaseModel()on every short-lived provider dispose if they reconnect often (e.g. BLE flaps); release when leaving on-device STT entirely.Test plan
releaseModel()then transcribe again; confirm reload happens once