OpenAI realtime plugin throws on `agent_config_update`: post-interruption context prune and WS reconnect both fail

[JoshPearre]

Summary

The OpenAI realtime plugin cannot serialize agent_config_update items, but AgentActivity inserts them into agent._chatCtx and then hands that same context to the plugin. Every code path that replays the chat context through RealtimeSession.createChatCtxUpdateEvents() therefore throws Unsupported item type: agent_config_update.

Two of those paths matter:

1. The post-interruption prune silently fails. After a caller barges in, assistant messages that were never played (played === 'skipped') are supposed to be pruned from the model's context. The prune throws, is swallowed as a logger.warn, and the realtime session keeps assistant text the user never heard. The model then behaves as if it had said things it never said.
2. The WebSocket reconnect fails, after destroying remoteChatCtx. Here the throw is not caught by the adjacent try/catch, and it happens after this.remoteChatCtx has already been replaced with a fresh RemoteChatContext() — so the this.remoteChatCtx = oldChatCtx rollback never runs.

We hit (1) in production on gpt-realtime telephony calls. After any barge-in the model's context contained text the caller never heard, which surfaced as the agent confidently referring to things it had never said.

Versions

• @livekit/agents 1.4.5, @livekit/agents-plugin-openai 1.4.5
• Still present in 1.4.11 and 1.5.0 (checked the published tarballs). 1.5.0 adds Agent.updateInstructions(), which records another AgentConfigUpdate — so the exposure grows.

Mechanism

AgentActivity creates and inserts the item (line numbers from the published dist/):

// agents/dist/voice/agent_activity.js:327  (initial config)
// agents/dist/voice/agent_activity.js:545  (tools added/removed)
const configUpdate = new AgentConfigUpdate({ toolsAdded, toolsRemoved });
this.agent._chatCtx.insert(configUpdate);

It then passes the raw agent._chatCtx to the plugin after an interruption:

// agents/dist/voice/agent_activity.js:2393
if (anySkipped && realtimeModel.capabilities.midSessionChatCtxUpdate) {
  try {
    await realtimeSession.updateChatCtx(this.agent._chatCtx);   // <-- contains agent_config_update
  } catch (error) {
    this.logger.warn({ error }, ...);                           // <-- always taken; prune never happens
  }
}

The plugin copies the context without excluding config updates, then maps every item:

// agents-plugin-openai/dist/realtime/realtime_model.js:453
async createChatCtxUpdateEvents(chatCtx, addMockAudio = false) {
  const newChatCtx = chatCtx.copy();     // <-- keeps agent_config_update
  ...
}

// agents-plugin-openai/dist/realtime/realtime_model.js:1542, in livekitItemToOpenAIItem()
default:
  throw new Error(`Unsupported item type: ${item.type}`);

The reconnect path has the same defect, and it is the more dangerous one:

// agents-plugin-openai/dist/realtime/realtime_model.js:759
const chatCtx = this.chatCtx.copy({
  excludeFunctionCall: true,
  excludeInstructions: true,
  excludeEmptyMessage: true,
  // no excludeConfigUpdate
});
const oldChatCtx = this.remoteChatCtx;
this.remoteChatCtx = new llm.RemoteChatContext();        // state destroyed here
events.push(...await this.createChatCtxUpdateEvents(chatCtx));   // <-- throws, OUTSIDE the try
try {
  for (const ev of events) { wsConn.send(JSON.stringify(ev)); }
} catch (error) {
  this.remoteChatCtx = oldChatCtx;                        // rollback never reached
  throw new APIConnectionError({ ... });
}

So any reconnect on a session whose agent has ever had a tool added/removed (or, in 1.5.0, whose instructions were updated) fails with Unsupported item type and leaves remoteChatCtx empty.

Reproduction

1. Build a realtime agent with tools (so AgentActivity inserts an AgentConfigUpdate when tools are registered).
2. Start a session with gpt-realtime (capabilities.midSessionChatCtxUpdate === true).
3. Let the agent begin a long reply, then interrupt it mid-sentence so at least one message output ends up played === 'skipped'.
4. Observe the warning from agent_activity.js:2394: Unsupported item type: agent_config_update. The skipped assistant text remains in the realtime session's context.

Suggested fix

The SDK already knows this item must be excluded before it reaches a provider — agent_activity.js:405,409 does exactly this:

this.realtimeSession.chatCtx.copy({
  excludeInstructions: true,
  excludeHandoff: true,
  excludeConfigUpdate: true,
})

So the one-line fix is to do the same in the plugin, which covers both call paths at once:

  async createChatCtxUpdateEvents(chatCtx, addMockAudio = false) {
-   const newChatCtx = chatCtx.copy();
+   const newChatCtx = chatCtx.copy({ excludeConfigUpdate: true });

Alternatively (or additionally), livekitItemToOpenAIItem() could skip rather than throw on item types the provider has no representation for, and the reconnect path could be reordered so remoteChatCtx is only replaced after the events are built successfully.

We are running the createChatCtxUpdateEvents fix as a patch-package patch in production; it resolved the corrupted-context behavior (zero agent_config_update errors since, across live calls with barge-ins).

Happy to open a PR if that shape looks right.