Add Deepgram Voice Agent framework support#144
Conversation
Adds a `deepgram` assistant-server framework backed by Deepgram's Voice Agent API (unified STT->LLM->TTS over a single WebSocket), so it can be benchmarked like the existing S2S frameworks. - New DeepgramAssistantServer (src/eva/assistant/deepgram_server.py), modeled on the Gemini Live server and the assistant_server_contract. - Register `deepgram` in worker._get_server_class and the framework Literal. - Parse raw WebSocket JSON by event `type` rather than the SDK's typed iterator, which in deepgram-sdk 6.1.x mis-deserializes every agent event as the same model (dropping transcripts and tool-call requests). - KeepAlive task to prevent Deepgram's ~10s input-audio timeout from closing the session while the (half-duplex) agent is speaking. - Compute model_response latency from the server-side receipt time of user_speech_stop (the simulator emits it on a monotonic clock). - Unit tests for settings/tool conversion + framework dispatch test. - Docs section in assistant_server_contract.md; .env.example framework enum. - Bump simulation_version 2.0.0 -> 2.1.0 (affects benchmark outputs).
…agent-framework # Conflicts: # src/eva/__init__.py
Co-authored-by: Katrina Stankiewicz <katrina.stankiewicz@servicenow.com>
Builds on the code-review suggestions (model now required in s2s_params;
use base self.language):
- Evaluate Deepgram as a cascade pipeline (get_pipeline_type -> CASCADE) since
it runs STT->LLM->TTS internally, so stt_wer / transcription_accuracy /
speakability run; expose {stt, llm, tts} via pipeline_parts so the run_id
folder shows the three component models.
- Add optional `think_label` to decouple the short metrics/run_id label from the
(long) Deepgram model id (Deepgram still receives `model`).
- Fix test broken by the review suggestion: _bare_server set `_language` but the
server now reads base `self.language`.
- Docs updated to match.
Add bring-your-own credentials/endpoint for the Voice Agent think step (s2s_params.think_credentials / think_endpoint / think_params / context_length / think_region) so the LLM can run on your own Bedrock/Anthropic/etc. quota instead of Deepgram's managed allowance — needed for long prompts that exceed managed limits and for an apples-to-apples comparison with the cascade pipeline. Managed config is unchanged; BYO is fully opt-in. For aws_bedrock, IAM/STS creds are built from AWS_* env vars when think_credentials is omitted. Note: Deepgram's aws_bedrock think integration only supports Claude 3.5-era models (e.g. anthropic/claude-3-5-haiku-20240307-v1:0). Haiku 4.5 is rejected with 'Invalid agent.think settings - model not available', so BYO Bedrock cannot match the cascade pipeline's Bedrock Haiku 4.5 — use managed or BYO Anthropic-direct (claude-haiku-4-5) for Haiku 4.5. Surface think outages loudly: a provider Error, or a greeting-only conversation (caller spoke but the model never replied), is now logged at ERROR with guidance instead of silently producing a clean-looking 1-turn run. Handle Welcome/SettingsApplied/History/AgentThinking events explicitly. Bumps simulation_version 2.0.2 -> 2.0.3.
…k settings The _bare_server() helper bypasses __init__, so it needed the new BYO attributes (_think_credentials/_think_endpoint/_think_params/_context_length/ _think_region) set explicitly. Also add coverage for BYO: bedrock credentials (explicit + AWS_* env-built), endpoint/params/context_length passthrough, and that the managed default is unchanged.
Custom Google endpoints require the model in the URL (.../models/<model>:streamGenerateContent) and reject it in settings, but the SDK's typed AgentV1Settings *requires* provider.model. Detect this case (think_provider=google + think_endpoint), omit model from the provider, and send the raw settings dict via the SDK's low-level _send to bypass the typed-model validation. Non-Google paths are unchanged (still validated + send_settings). Verified live: model-less google + streamGenerateContent drives multi-turn think. Bumps simulation_version 2.0.3 -> 2.0.4.
Log the Welcome event's request_id at INFO so each conversation can be traced in Deepgram's server-side logs / support tickets. Bumps simulation_version 2.0.4 -> 2.0.5.
| [[package]] | ||
| name = "deepgram-sdk" | ||
| version = "7.3.0" | ||
| version = "7.3.1" |
There was a problem hiding this comment.
Considering this version, does this comment from the PR description still apply?
SDK workaround: deepgram-sdk 6.1.x mis-deserializes every agent event to the same model via its typed iterator, dropping transcripts and tool-call requests. The server iterates the raw WebSocket and dispatches on the JSON type instead (audio still arrives as binary frames).
| # Bump simulation_version when changes affect benchmark outputs (agent code, | ||
| # user simulator, orchestrator, simulation prompts, agent configs, tool mocks). | ||
| simulation_version = "2.0.1" | ||
| simulation_version = "2.0.5" |
There was a problem hiding this comment.
The PR description says this 🤔
Version:
simulation_version2.0.0 → 2.1.0 (affects benchmark outputs).
Anyway, I'm not sure we need to bump the version. We don't need to rerun any old simulations, right? This addition doesn't really "affect" existing results; it just adds new possibilities. Does that make sense?
| simulation_version = "2.0.5" | |
| simulation_version = "2.0.1" |
| "stt": p.get("listen_model", "nova-3"), | ||
| "llm": p.get("think_label") or p.get("model") or p.get("think_model", ""), | ||
| "tts": p.get("speak_model", "aura-2-thalia-en"), |
There was a problem hiding this comment.
I'd like to double-check, maybe with @fanny-riols, whether it makes sense to have default models, which will certainly go out of fashion, or if it would be better to require the user to specify the models.
If the default models are OK, we should at least reuse the constants here:
| "stt": p.get("listen_model", "nova-3"), | |
| "llm": p.get("think_label") or p.get("model") or p.get("think_model", ""), | |
| "tts": p.get("speak_model", "aura-2-thalia-en"), | |
| "stt": p.get("listen_model", _DEFAULT_LISTEN_MODEL), | |
| "llm": p.get("think_label") or p.get("model") or p.get("think_model", ""), | |
| "tts": p.get("speak_model", _DEFAULT_SPEAK_MODEL), |
There was a problem hiding this comment.
Yeah I think it's fine to have defaults for the deepgram framework. But let's use the DEFAULT vars though.
| p = self.s2s_params or {} | ||
| return { | ||
| "stt": p.get("listen_model", "nova-3"), | ||
| "llm": p.get("think_label") or p.get("model") or p.get("think_model", ""), |
There was a problem hiding this comment.
This "think_model" is not used anywhere else. Was that a confusion between "think_label" and "think_model"? Or am I missing something?
Also, is there a reason to use that instead of "alias" like in regular S2S models? If we can go with "alias", then we can reuse this function:
| "llm": p.get("think_label") or p.get("model") or p.get("think_model", ""), | |
| "llm": _param_alias(p), |
And adapt DeepgramAssistantServer.__init__() accordingly.
| p = self.s2s_params or {} | ||
| return { | ||
| "stt": p.get("listen_model", "nova-3"), | ||
| "llm": p.get("think_label") or p.get("model") or p.get("think_model", ""), | ||
| "tts": p.get("speak_model", "aura-2-thalia-en"), | ||
| } |
There was a problem hiding this comment.
Thanks to the _check_companion_services() model validator, we can assume that self.s2s_params is a dict (not None), so self.s2s_params or {} is not necessary.
Combining that with the two suggestions above, we get this. Do all these make sense?
| p = self.s2s_params or {} | |
| return { | |
| "stt": p.get("listen_model", "nova-3"), | |
| "llm": p.get("think_label") or p.get("model") or p.get("think_model", ""), | |
| "tts": p.get("speak_model", "aura-2-thalia-en"), | |
| } | |
| return { | |
| "stt": self.s2s_params.get("listen_model", _DEFAULT_LISTEN_MODEL), | |
| "llm": _param_alias(self.s2s_params), | |
| "tts": self.s2s_params.get("speak_model", _DEFAULT_SPEAK_MODEL), | |
| } |
Summary
Adds a new
deepgramassistant-server framework backed by Deepgram's Voice Agent API (unified STT→LLM→TTS over a single WebSocket), so the Deepgram agent can be benchmarked like the existing S2S frameworks. Modeled on the Gemini Live server and thedocs/assistant_server_contract.mdcontract.Changes
src/eva/assistant/deepgram_server.py(DeepgramAssistantServer): bridges the Twilio-framed user-simulator WebSocket ↔client.agent.v1.connect(), with the standard audit-log / framework-log / metrics-log / audio-buffer handling.deepgramadded toworker._get_server_class()and theframeworkLiteralinmodels/config.py.tests/unit/assistant/test_deepgram_server.py(settings + tool-conversion) and a dispatch case intest_framework_dispatch.py.assistant_server_contract.md;deepgramadded to the framework enum in.env.example.simulation_version2.0.0 → 2.1.0 (affects benchmark outputs).Implementation notes (found via live testing)
deepgram-sdk6.1.x mis-deserializes every agent event to the same model via its typed iterator, dropping transcripts and tool-call requests. The server iterates the raw WebSocket and dispatches on the JSONtypeinstead (audio still arrives as binary frames).KeepAlivetask prevents Deepgram's ~10s input-audio timeout from closing the session while the half-duplex agent is speaking.model_responselatency is measured from the server-side receipt time of the simulator'suser_speech_stopevent.Testing
ruff,mypy(new file clean), andpre-commit run --all-filesall pass (version-bump + metric-signature hooks included).conversation_valid_end = 1.0), 500+ tool calls executed with scenario-DB mutations, and zero framework-level errors (no disconnects, audio timeouts, or event-processing failures).Config example
EVA_FRAMEWORK=deepgram EVA_MODEL__S2S=deepgram EVA_MODEL__S2S_PARAMS='{"api_key":"<deepgram-key>","model":"gpt-4o-mini"}'Optional
s2s_params(defaults):think_provider(open_ai),listen_model(nova-3),speak_model(aura-2-thalia-en),language(en).