feat: voice input (dictation) using Claude CLI speech-to-text - #25
Merged
Conversation
Adds a microphone toggle to the chat toolbar that dictates into the input, mimicking the Claude Code CLI's `/voice`. The CLI's voice is TUI-only and unreachable from our headless `claude -p` stream-json driver, so we replicate its wire protocol directly: - Stream 16kHz mono linear16 PCM to the private WebSocket endpoint wss://api.anthropic.com/api/ws/speech_to_text/voice_stream (a Deepgram Nova-3 proxy), authenticated with the Claude.ai OAuth token already in ~/.claude/.credentials.json — the same token/path the CLI uses. - Audio captured from an auto-detected system recorder (parec / arecord / rec / ffmpeg), the same fallbacks the CLI relies on. - KeepAlive every 8s, CloseStream + trailing-final drain on stop; interim transcripts preview live in the input and commit on TranscriptEndpoint. Threading follows the codebase convention: a worker thread owns the blocking tungstenite socket and emits VoiceEvents over mpsc; the UI polls via timeout_add_local. No async runtime. New deps: tungstenite (rustls/ring). Caveat (documented in the module): this is an undocumented endpoint reached with subscription credentials, so it can break on any CLI release and is not a supported API. All brittle assumptions are isolated in services/voice.rs. Handshake/auth validated live against the server; 5 unit tests cover the transcript message parser.
SergKam
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a microphone toggle to the chat toolbar that dictates speech into the input box, mimicking the Claude Code CLI's
/voice.The CLI's voice dictation is a TUI-only feature and is unreachable from our headless
claude -p --input-format stream-jsondriver (no flag, no stream-json message, no env var exposes it). So rather than delegate, this replicates the CLI's exact wire protocol — reverse-engineered from the v2.1.175 bundle and validated live (handshake + auth accepted by the server).How it works
linear16PCM (16 kHz, mono) to the private WebSocketwss://api.anthropic.com/api/ws/speech_to_text/voice_stream(an Anthropic-proxied Deepgram Nova-3), with the Deepgram query params the CLI sends.Authorization: Bearer <accessToken>read from~/.claude/.credentials.json(claudeAiOauth) — the same token and path the CLI uses. Plus the CLI'sUser-Agent/x-app/anthropic-client-platformheaders to pass the Cloudflare edge.parec→arecord→rec(SoX) →ffmpeg— the same fallbacks the CLI uses. Friendly error if none is installed.{"type":"KeepAlive"}on open + every 8s;{"type":"CloseStream"}on stop, then drains trailing finals.TranscriptInterim/TranscriptTextpreview live in the input;TranscriptEndpointcommits the segment.UX
Architecture
Follows the codebase convention — a worker thread owns the blocking
tungstenitesocket and emitsVoiceEvents overmpsc; the UI polls viatimeout_add_local. No async runtime. New deps:tungstenite(rustls +ring).This is an undocumented endpoint reached with subscription credentials. It can break on any CLI release and is not a supported API; using subscription auth from a third-party client is plausibly against ToS. All brittle assumptions are isolated in
src/services/voice.rs. Chosen deliberately by the maintainer over the robust local-Whisper / configurable-command alternatives.Tests