feat(telnyx): add Telnyx assistant server framework#173
Open
lucasassisrosa wants to merge 11 commits into
Open
feat(telnyx): add Telnyx assistant server framework#173lucasassisrosa wants to merge 11 commits into
lucasassisrosa wants to merge 11 commits into
Conversation
Add Telnyx hosted AI Assistant as a first-class EVA framework alongside
the existing Pipecat pipeline. The Telnyx provider connects to the Telnyx
AI gateway via direct anonymous WebRTC (Verto JSON-RPC over WebSocket),
bridges audio between the EVA user simulator (Twilio mulaw frames) and the
Telnyx assistant (PCM 16 kHz), and enriches the audit log from the Telnyx
Conversations API after the call ends.
New files:
- src/eva/assistant/base_server.py
AbstractAssistantServer: template-method base class that defines the
EVA server contract (FastAPI WebSocket endpoint, audio bridging, tool
execution via ToolExecutor, audit log, audio recording, save_outputs).
Concrete servers implement start() and _shutdown(); the base handles
stop(), output saving, and artifact generation.
- src/eva/assistant/audio_bridge.py
Audio format conversion utilities: mulaw 8 kHz ↔ PCM 16-bit 24 kHz
resampling (libsoxr), PCM mixing, Twilio FrameSerializer protocol
helpers, and framework/metrics log writers.
- src/eva/assistant/telnyx_server.py
TelnyxAssistantServer(AbstractAssistantServer): the main server.
- TelnyxAssistantManager: REST CRUD for benchmark assistants.
- TelnyxDirectSessionConfig: dataclass for WebRTC session params.
- TelnyxVertoJsonRpcClient: Verto JSON-RPC over WebSocket.
- TelnyxWebRTCClient: aiortc RTCPeerConnection + media management.
- TelnyxAssistantServer: bridges EVA Twilio frames to Telnyx WebRTC,
handles conversation events, tool calls, dynamic variables,
audit-log enrichment from Conversations API.
- src/eva/utils/audio_utils.py
save_pcm_as_wav() helper shared by all server implementations.
- tests/unit/assistant/test_telnyx_server.py (20 tests)
- tests/unit/assistant/test_audio_bridge.py (3 tests)
Modified:
- src/eva/models/config.py: Add framework field to RunConfig.
- src/eva/orchestrator/worker.py: Add _get_server_class() dispatch.
- pyproject.toml: Add aiortc, aiohttp, soxr dependencies.
Addresses VSDK-277 review: adds the missing Python EVA wrapper that
subclasses AbstractAssistantServer, registers in worker.py, routes tool
calls through ToolExecutor, and produces all required EVA artifacts.
Add Telnyx hosted AI Assistant as a first-class EVA framework alongside
the existing Pipecat pipeline. The Telnyx provider connects to the Telnyx
AI gateway via direct anonymous WebRTC (Verto JSON-RPC over WebSocket),
bridges audio between the EVA user simulator (Twilio mulaw frames) and the
Telnyx assistant (PCM 16 kHz), and enriches the audit log from the Telnyx
Conversations API after the call ends.
New files:
- src/eva/assistant/base_server.py
AbstractAssistantServer: template-method base class that defines the
EVA server contract (FastAPI WebSocket endpoint, audio bridging, tool
execution via ToolExecutor, audit log, audio recording, save_outputs).
Concrete servers implement start() and _shutdown(); the base handles
stop(), output saving, and artifact generation.
- src/eva/assistant/audio_bridge.py
Audio format conversion utilities: mulaw 8 kHz ↔ PCM 16-bit 24 kHz
resampling (libsoxr), PCM mixing, Twilio FrameSerializer protocol
helpers, and framework/metrics log writers.
- src/eva/assistant/telnyx_server.py
TelnyxAssistantServer(AbstractAssistantServer): the main server.
- TelnyxAssistantManager: REST CRUD for benchmark assistants.
- TelnyxDirectSessionConfig: dataclass for WebRTC session params.
- TelnyxVertoJsonRpcClient: Verto JSON-RPC over WebSocket.
- TelnyxWebRTCClient: aiortc RTCPeerConnection + media management.
- TelnyxAssistantServer: bridges EVA Twilio frames to Telnyx WebRTC,
handles conversation events, tool calls, dynamic variables,
audit-log enrichment from Conversations API.
- src/eva/utils/audio_utils.py
save_pcm_as_wav() helper shared by all server implementations.
- tests/unit/assistant/test_telnyx_server.py (20 tests)
- tests/unit/assistant/test_audio_bridge.py (3 tests)
Modified:
- src/eva/models/config.py: Add framework field to RunConfig.
- src/eva/orchestrator/worker.py: Add _get_server_class() dispatch.
- pyproject.toml: Add aiortc, aiohttp, soxr dependencies.
Addresses VSDK-277 review: adds the missing Python EVA wrapper that
subclasses AbstractAssistantServer, registers in worker.py, routes tool
calls through ToolExecutor, and produces all required EVA artifacts.
lucasassisrosa
force-pushed
the
feat/telnyx-framework-provider
branch
from
7376bea to
fdb747f
Compare
Add the TURNS/443 fallback (turns:turn2.telnyx.com:443) to TELNYX_DEFAULT_ICE_SERVERS so the Telnyx EVA server's prod ICE list matches DEFAULT_PROD_ICE_SERVERS in @telnyx/webrtc 2.27.4: STUN + TURN UDP/3478 + TURN TCP/3478 + TURNS on 443. This is the last-resort TLS fallback for networks that block both 3478 transports.
The direct-WebRTC session could establish signaling + ICE + DTLS
(iceConnectionState=completed, connectionState=connected) yet receive
zero remote audio frames, leaving both audio buffers empty and the
conversation silent until the far end idle-disconnected ~90s later — with
no diagnostic trace.
- Log iceConnectionState / connectionState transitions.
- On connectionState failed|closed, set disconnected_event so the session
ends immediately instead of idling.
- In _receive_remote_audio, bound track.recv() with a 10s timeout and warn
("media not flowing", with iceConnectionState + frame count) instead of
blocking silently; log the first received frame and frame count on stop.
Diagnostic/robustness only — does not change the audio bridge behaviour
when media flows.
When s2s_params contains connection_id + webhook_base_url, use TelnyxCallControlTransport (REST + media streaming WebSocket) instead of TelnyxWebRTCClient (aiortc/SRTP). This bypasses the unresolved SRTP unprotect failure in aiortc. Changes: - Add _use_call_control property to TelnyxAssistantServer - Branch in _handle_session: Call Control when params present, WebRTC otherwise (preserves existing behavior) - Move _conversation_event_handler assignment to WebRTC branch only (Call Control transport doesn't use it) - Extend _validate_runtime_config with Call Control credential validation (api_key, connection_id, from_number, to, webhook_base_url) - Add 2 unit tests: Call Control missing credentials, all fields present - Update 2 existing tests to set _webhook_base_url (needed by new property) Blocker ServiceNow#1 from the live benchmark readiness checklist.
…y writes _append_user_events_from_simulator() and _enrich_audit_log_from_external_sources() hardcoded "elevenlabs_events.jsonl". That is upstream's LEGACY artifact name (LEGACY_ELEVENLABS_EVENTS_FILENAME); the user simulator writes "user_simulator_events.jsonl" (user_simulator/base.py). The legacy file therefore never exists at runtime, extract_user_speech_events() always returned [], and the audit log ended up with no user turns at all -- so a graded run would score a conversation missing its entire user side. Use resolve_user_simulator_events_path(), the upstream helper that prefers the neutral filename and still falls back to the legacy one, and warn when neither exists instead of silently producing an empty transcript. The unit test masked this by writing the legacy filename as its fixture; it is now parametrized over both names, so the real pipeline filename is actually exercised. Also sorts the eva.utils/eva.models imports, which ruff flagged.
The direct-WebRTC session reached iceConnectionState=completed and
connectionState=connected, and aiortc logged "DTLS handshake complete" -- yet no
audio moved in either direction. The assistant generated its greeting (visible in
ai_conversation events) but we received 0 RTP frames, and 5s of real speech sent
upstream produced "[long silence]" and no user transcript.
Root cause is the DTLS final flight. OpenSSL's DTLS server answers the client's
Finished with NewSessionTicket + ChangeCipherSpec + Finished, and aiortc never sets
SSL_OP_NO_TICKET. The ticket is ~1.4 kB, so it is fragmented across several DTLS
records. Telnyx's FreeSWITCH does not process the fragmented ticket and therefore
never consumes our Finished: it re-sent its entire final flight 5 times
(Certificate/ClientKeyExchange/CertificateVerify/CCS/Finished), never completed the
handshake, and never derived SRTP keys. OpenSSL considers the handshake complete on
our side, so aiortc reports "connected" while media is silently dead BOTH ways --
FreeSWITCH cannot decrypt our audio and will not send us its own.
Set SSL_OP_NO_TICKET on the DTLS context. Session resumption is meaningless for a
one-shot DTLS-SRTP call and browsers do not offer tickets here either, so the final
flight collapses to an unfragmented CCS + Finished.
Measured against assistant-473093df (Pre-Authorization Support), 20s call:
before after
DTLS datagrams 27 3 (retransmit storm -> clean handshake)
SRTP packets 8-15 814
inbound audio frames 0 790
aiortc exposes no hook for SSL options (RTCConfiguration carries only iceServers and
bundlePolicy), so this patches RTCCertificate._create_ssl_context once, idempotently.
…startup
Running EVA the documented way (uv sync --all-extras; cp .env.example .env;
EVA_FRAMEWORK=telnyx python main.py) surfaced a shutdown race that the direct-transport
tests never hit.
TelnyxWebRTCClient.stop() sets self._peer = None. If the session is torn down while
start() is still awaiting ICE gathering -- which is exactly what happens when the user
simulator fails to come up, e.g. a bad OPENAI_API_KEY for the caller -- start() resumes
and dereferences the now-None peer:
ERROR | eva.assistant.telnyx_server | Telnyx session error:
'NoneType' object has no attribute 'localDescription'
That masks the real failure (the caller never started) behind a confusing AttributeError,
and it fires once per retry attempt.
Check _peer after the ICE-gathering await and exit quietly if stop() already ran: there is
no session left to send an invite for.
INFO | eva.assistant.telnyx_server | Telnyx WebRTC session stopped during startup; skipping invite
…ts a turn
Found by running the EVA CLI properly (EVA_FRAMEWORK=telnyx python main.py) rather than
driving the transport directly: record 1.1 failed with
Record 1.1: Extracted turns - 1 assistant (keys [0]), 0 user (keys [])
Skipping record 1.1: no assistant or user turns found
and audio_user.wav was 151s of digital zero -- the caller never spoke at all.
The user simulator treats EVERY media frame as assistant speech. audio_bridge's
_receive_from_assistant() never inspects the payload: any non-empty frame calls
_on_assistant_audio_start() and refreshes _assistant_audio_ended_time, and the turn is
considered over only after 200 ms with NO frames arriving.
Telnyx's WebRTC leg streams RTP continuously (~50 pps) for the whole call, so between turns
we decode frames of pure silence -- and we forwarded them. The simulator therefore received
a media frame every 20 ms for the entire call and concluded the assistant never stopped
talking: is_assistant_playing() stayed true, _assistant_turn_is_settled() never became true,
the caller's response task waited forever (8x caller_response_coalesced, caller_response_created
never logged), and the record died with zero user turns.
Drop frames whose RMS is at or below ASSISTANT_SILENCE_RMS so the simulator sees a real gap.
EVA CLI, same command, before and after:
before after
extracted turns 1 assistant, 0 user 6 assistant, 6 user
audio_user_clean.wav 0.0s (caller never spoke) 39.9s of real speech
The record still fails, but now for a different and clearly external reason: the assistant
does not receive the caller's audio at all. It re-asks a question the caller has already
answered and falls into its no-input script -- consistent with Telnyx's own dual-channel
recording, where the caller track is digital silence. That is a Telnyx-side media bug
(VSDK-277), not an EVA-contract bug.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Telnyx hosted AI Assistant as a first-class EVA framework alongside the existing Pipecat pipeline. The Telnyx provider connects to the Telnyx AI gateway via direct anonymous WebRTC (Verto JSON-RPC over WebSocket), bridges audio between the EVA user simulator (Twilio mulaw frames) and the Telnyx assistant (PCM 16 kHz), and enriches the audit log from the Telnyx Conversations API after the call ends.
New files:
src/eva/assistant/base_server.py AbstractAssistantServer: template-method base class that defines the EVA server contract (FastAPI WebSocket endpoint, audio bridging, tool execution via ToolExecutor, audit log, audio recording, save_outputs). Concrete servers implement start() and _shutdown(); the base handles stop(), output saving, and artifact generation.
src/eva/assistant/audio_bridge.py Audio format conversion utilities: mulaw 8 kHz ↔ PCM 16-bit 24 kHz resampling (libsoxr), PCM mixing, Twilio FrameSerializer protocol helpers, and framework/metrics log writers.
src/eva/assistant/telnyx_server.py TelnyxAssistantServer(AbstractAssistantServer): the main server. - TelnyxAssistantManager: REST CRUD for benchmark assistants. - TelnyxDirectSessionConfig: dataclass for WebRTC session params. - TelnyxVertoJsonRpcClient: Verto JSON-RPC over WebSocket. - TelnyxWebRTCClient: aiortc RTCPeerConnection + media management. - TelnyxAssistantServer: bridges EVA Twilio frames to Telnyx WebRTC, handles conversation events, tool calls, dynamic variables, audit-log enrichment from Conversations API.
src/eva/utils/audio_utils.py save_pcm_as_wav() helper shared by all server implementations.
tests/unit/assistant/test_telnyx_server.py (20 tests)
tests/unit/assistant/test_audio_bridge.py (3 tests)
Modified:
Addresses VSDK-277 review: adds the missing Python EVA wrapper that subclasses AbstractAssistantServer, registers in worker.py, routes tool calls through ToolExecutor, and produces all required EVA artifacts.