Voice: record spoken replies in the chat and let the agent work while audio plays

[eddiemachaalani]

Problem

Two voice-mode failure modes, both hit in real use:

1. The agent freezes while it talks. The speak handler awaited playback confirmation from the client, so the agent could not start any work until the audio finished playing on the phone. The observed flow was: think → speak → silence while the audio plays → then the first tool call. For a multi-sentence acknowledgment that's many seconds of dead time before work begins.
2. Voice fails silently. The transcript entry for a spoken reply was only emitted after playback completed. When synthesis or playback failed (TTS provider error, client never confirming), the reply vanished entirely — the user heard nothing and had nothing to read either.

Change

speak is now "a chat message that also plays as audio":

• The assistant transcript entry is emitted immediately, before any audio work — the chat always carries the reply as a durable record and readable fallback.
• Playback is queued on a per-session promise chain: overlapping speak calls play in spoken order, and a barge-in drains the queue through the existing abort path.
• The tool returns once the utterance is queued, so the agent keeps working while audio plays.
• Synthesis/playback failure surfaces as an error activity entry pointing at the text ("The reply is shown as text above.").
• The voice-mode prompt now tells the model the spoken text is recorded automatically (so it stops duplicating replies as normal messages) and to continue working while audio plays. The tool description is updated to match.

QA evidence

Live on a daemon running this branch, driven from the iOS app: the acknowledgment audio plays while the agent's shell commands are already streaming — previously work only started after the audio ended. Text for every utterance appears in the chat at speak time.

Tests — 4 new, in voice-session.test.ts:

npx vitest run src/server/session/voice/voice-session.test.ts
 Test Files  1 passed (1)
      Tests  8 passed (8)

• records the spoken text in the chat before audio and returns without awaiting playback
• plays overlapping speak calls in order, starting the next only after playback confirms
• drops queued speech after an abort but keeps the text record
• surfaces synthesis failure as an error entry while keeping the text record

Mutation-checked: restoring the old blocking behavior (await this.speechQueue in the handler) turns 3 of them red (timeouts, exactly the old freeze); reverting goes green. Typecheck and lint clean.

Platforms

Server-only change. Tested on a macOS daemon end-to-end from the iOS app. Not separately tested: Android app (no client change), Windows/Linux daemons (no platform-specific code involved).

Happy to adjust semantics (e.g., a queue-depth cap on pending utterances) if you'd rather different behavior here.

🤖 Generated with Claude Code

[boudra]

Closing in favor of the app-wide voice implementation in #4157, which is still open.