fix(desktop): transcribe only the local microphone in Huddles - #7043
Open
hr-o wants to merge 1 commit into
Open
Conversation
Each Huddle transcript is a kind:9 event. The local user signs this event. Thus, do not send audio from other participants to the STT pipeline. If the desktop transcribes remote audio, it incorrectly identifies the remote speaker as the local user. This error occurs on each desktop that receives the audio. Remove the remote-human transcription path. Do not filter remote transcripts before publication. The STT pipeline now receives audio only from the local microphone. This design also prevents Parakeet from processing remote audio that the desktop cannot publish correctly. Known regression: legacy text-only agents no longer receive transcripts of speech from mobile participants. These agents cannot respond to that speech until an agent audio runtime is available. An agent that joins the Opus room directly does not depend on transcripts and is therefore unaffected, but no such agent exists yet. Signed-off-by: hr-o <126079238+hr-o@users.noreply.github.com>
🔐 Codex Security Review
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related work
No open PR or issue fixes this. Closest existing work, none of which overlaps the fix itself:
fix(desktop): drop punctuation-only STT transcripts in huddles— same publish pathProblem
A Huddle transcript is published as a kind:9 signed with the local user's key. But
playout.rsfed every remote human peer's audio into the same STT pipeline: each active peer slot that wasn't a locally-synthesized TTS publisher or a known agent was summed into one mono buffer bymix_remote_stt_samplesand pushed in viapush_remote_audio.The result is forged provenance. When Alice speaks, Bob's desktop transcribes her and publishes her words as a message signed by Bob — independently, on every listening desktop in the room. There is no diarization and no host election, so a three-desktop Huddle turns one sentence from Alice into a kind:9 from Bob and another from Carol, each claiming to be that user's own speech.
The remote path was also deliberately ungated by push-to-talk and mute, so a listener who never unmuted still published other people's speech under their own key.
Fix
Remove the producer rather than filter at the publisher.
The alternative was to carry an origin through the text layer —
TranscriptSegment { origin, text }— and admit onlyLocalMicrophonebefore publishing. But onceplayout.rsstops feeding remote audio in, that origin has exactly one possible value:RemoteHumanbecomes an unconstructed variant,push_remote_audioa method with no caller, and any test would assert an unreachable branch. So the invariant is expressed by deleting the producer and documenting it on the single surviving entry point.Secondary benefit: every listening desktop stops running Parakeet over audio it must discard.
Gone:
SttAudioOriginand its second VAD stream,push_remote_audio,track_local_floor,mix_remote_stt_samples,f32_samples_to_le_bytes, the mix block in the playout receive loop, and theremote_stt_pipelineweak handle inHuddleState. The STT channel now carries plain PCM. Net −152 lines across 5 files, all underdesktop/src-tauri/src/huddle/.No relay change. No protocol change.
Known regression
Legacy text-only agents lose the accidental proxy by which mobile speech reached them, and stay silent until an agent audio runtime lands.
Mobile has no local STT, so a phone's speech reached text-driven agents only because some desktop in the room transcribed it and published it under that desktop user's key. That path is now gone, and nothing replaces it in this PR.
Speech-to-speech agents are unaffected — they receive the Opus room directly.
This is a deliberate trade: correct attribution over an undesigned capability. Publishing one participant's speech under another participant's signature is forged provenance in a signed-event system, and the lost behaviour was a side effect of every listener transcribing the room, not a designed feature.
Verification
Automated, on this branch's base:
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings— clean-D warnings— cleancargo test --workspace(Tauri) — 3004 passed / 0 failed / 18 ignoredjust cigreen: desktop JS 5799/0, desktop and web builds,flutter analyzeclean, mobile 2007 testsManual, on real hardware — one Windows desktop as a muted listener, one phone as the speaker, against a stock
ghcr.io/block/buzz:mainrelay:The listener's own transcript is the control that makes the zero meaningful: STT was running, VAD was flushing, NIP-98 authenticated and the relay accepted, 42 seconds after the phone joined the room. It simply had nothing to say about the phone's audio.
Why there is no new regression test
The usual shape — a test asserting that remote audio never reaches the publisher — has no code path left to exercise. Removing the producer means there is no way to construct the failing input:
push_remote_audiohas no caller,RemoteHumanhas no constructor, and the mixing function is gone. A test could only assert that a symbol no longer exists, which the compiler already enforces at every call site.So the guarantee is structural rather than behavioural. What the PR does instead:
SttPipeline::push_audio, the single surviving entry point, and in the module headerremote_agent_identity_is_excluded_from_the_human_sttto…_from_the_human_floor, which is what it actually asserts nowNot included
No UI change, so there are no screenshots.
Restoring speech input for text-only agents is deliberately out of scope. It belongs with an agent audio runtime that joins the Opus room directly, which is follow-up work.
The pre-fix run used the released 0.5.19 client, so the defect is reproducible in shipped code —
push_remote_audioarrived in #6056 on 2026-08-22, three days before that tag.