fix(drift): attribute a WS refusal, close the CLI signal race, and probe a modality Gemini serves - #374
Merged
Merged
Conversation
The CLI logged 'aimock server listening on <url>' and only then called
process.on('SIGTERM'). Writes to a pipe are synchronous on Linux, so a
supervisor that reacts to that line can deliver SIGTERM while the process is
still mid-statement. In that window the signal reaches Node's default
disposition, which re-raises it: the process dies with a null exit code and
never runs its own shutdown. Registering first closes the window.
Observed on linux/node20, freezing the child at the instant it announces
readiness: 55/60 iterations died with code=null signal=SIGTERM before the
change, 0/60 after.
The --watch reload test waited for 'listening on' and then asserted that 'Watching' was ALREADY in stdout. The watcher is started after the readiness line is logged, so the assertion raced the child; it failed 2 of 30 full-suite runs in a CI-shaped container (4 cpu / 16 GB, linux, node 20). Wait for the line that actually marks the watcher as up.
… by its test name
Run 31571018005 quarantined two Gemini Live legs as "could not be mapped to a
provider" (exit 5 — the manual-triage stop). Google had refused the session with
RFC 6455 code 1007 and stated the cause, and the collector recognized the close;
what failed was attribution. WS attribution keyed only off a `*.drift.ts` stack
frame, and the failure is raised by the SHARED client (ws-providers.ts) from a
socket callback, so the probe frame does not survive the async boundary — it is
absent from the stack, not merely deeper, and no frame pattern can recover it.
The failing test's name still carries the answer ("Gemini Live WS drift > …"),
so attribution now reads the stack frame FIRST and falls back to the suite title,
anchored at the start and matched against the SAME closed WS_HANDSHAKE_PROBES
table via each surface's registry provider label. Refusing to guess is unchanged:
a title matching no registered label yields null and the failure still
quarantines at exit 5.
The two production legs now attribute to Gemini Live / src/ws-gemini-live.ts as
critical drift (exit 2, non-fatal on the base leg) instead of hard-failing it.
…efix invariant The longest-label-first sort could not be made to fail: no registered WS probe label is a prefix of another, so no input distinguished the ordering. An ordering rule nothing can exercise is coverage that does not exist, so it is gone and the property it relied on is asserted instead — registering a prefix-colliding probe now turns a test RED rather than resolving by table order.
… stale build The CLI tests exercise dist/cli.js and `pnpm test` does not build, so a suite run right after a source change aims this guard at the previous build. It then reports the exact ungraceful-exit symptom it exists to detect, from a tree where the code is already correct. Assert the build is not older than src/cli.ts and say which command fixes it. Also widen the attempt loop from 3 to 12 — one attempt catches the unfixed ordering a little under half the time from inside the running suite, so three attempts miss it in roughly 18% of runs — and state in the name and the comment that the covered window starts at the readiness announcement: a SIGTERM delivered before then is still fatal.
The probe opened every Live session with responseModalities: ["TEXT"]. Every model exposing bidiGenerateContent is a native-audio model that supports only AUDIO, so Google refused the session out-of-band with an RFC 6455 close frame — code=1007, "The requested combination of response modalities (TEXT) is not supported by the model. models/gemini-3.1-flash-live-preview" — on every drift run since the listing began resolving to that model. A 1007 is a refusal, so the leg reported provider noise where drift should have been. Root cause was model SELECTION, not just the request: discovery required that the model name NOT contain "native-audio", using the name as a proxy for text-capability. gemini-3.1-flash-live-preview is a native-audio model whose name omits that substring, so the heuristic silently mis-classified it as text-capable and the probe asked it for TEXT. Selection now keys only on the listing's declared bidiGenerateContent support, and the leg drives AUDIO — the modality every Live model serves — grading the audio event sequence (inlineData parts + turnComplete) plus the modality-independent toolCall. Both sides of the three-way comparison send the same generationConfig, and the mock side is driven by an audio fixture so a real AUDIO turn is no longer compared against a mock TEXT one. The TEXT serverContent shape is not expressible against any live Live model; it stays covered mock-only by ws-gemini-live.test.ts. Drops the availability canary, whose body was expect(true).toBe(true) — it could not fail, so it reported health it never checked, and its premise (waiting for a text-capable model) is obsolete. ws-gemini-live-modality.test.ts drives the real probe against a local TLS server that enforces Google's modality rule, reproducing the 1007 frame verbatim from the drift-report artifact and running the whole three-way comparison locally, so the mock side is verified without live credentials.
commit: |
Contributor
Author
|
Merging with 25 other checks green; local suite 5280 passing; version-neutral. |
contextablemark
added a commit
that referenced
this pull request
Aug 19, 2026
…the release (#381) Adds the four user-facing `[Unreleased]` CHANGELOG entries that landed since the `v1.38.0` tag but had no entry, so the release notes are complete before the version is cut. Docs-only: `git diff --name-only origin/main..HEAD` is `CHANGELOG.md` and nothing else. Version-neutral — no bump, so it does not trigger `publish-release.yml`. Merge it before the release bump, or lift the text into the release commit; either works. Each entry was written from the actual commit diff, not the subject line: - **AG-UI `usage` on `RUN_FINISHED`/`RUN_ERROR`** — `aa68b26` (#378): the new optional `usage?: AGUITokenUsage[]`, typed and emitted, numeric-only, emitted only when supplied. - **Responses-API function-call collapse → `toolCalls`** — `23a8c3c` / #380: a tool-call-only Responses turn previously collapsed to empty content; now accumulates into `toolCalls` keyed by `output_index`, capturing `call_id`. - **Gemini Live speak-and-call ordering** — `a7d9b64` (#378): a `serverContent` (audio + text companion) strictly before the `toolCall`, then `turnComplete`; audio turns keep their companions. - **CLI signal handlers before readiness** — `1079b46` (#374): `SIGINT`/`SIGTERM` registered before the readiness log, closing the window where a supervisor's `SIGTERM` killed the process ungracefully (55/60 → 0/60). The residual — a signal strictly before readiness — is stated in the entry, drawn from code placement rather than a documented commit note. Verified: prettier clean, commitlint RC 0, no version-bearing file touched.
Merged
contextablemark
added a commit
that referenced
this pull request
Aug 19, 2026
Release cut for v1.39.0. No source changes. - Bumps root `package.json` version 1.38.0 → 1.39.0. - Converts the merged `[Unreleased]` material into a dated `## [1.39.0] - 2026-08-18` entry and opens a fresh empty `[Unreleased]`. All entries preserved in order (Added / Changed / Deprecated / Fixed), covering #382, #380, #378, #374, and the reset-canonicalization work (#358). Follow-up (not in this PR): `packages/aimock-pytest/src/aimock_pytest/_version.py` still pins `AIMOCK_VERSION = "1.38.0"`; bump it on the aimock-pytest release cadence per the changelog note.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups to #371. Three fixes: the drift collector could not attribute a WS refusal and stopped for manual triage; the CLI could be killed ungracefully by a signal arriving as it announced readiness; and the Gemini Live probe requested a modality its model does not serve.
1. A WS refusal now names its provider instead of stopping the run
After #371,
drift-live-prstill exited 5 — the message moved from "could not be parsed as drift reports" to "could not be mapped to a provider". The classification worked; attribution did not, and the manual-triage stop remained.From run
31571018005's own artifact, the two failures are not timeouts:1007is an RFC 6455 refusal, so the correct outcome is attributed critical drift at exit 2, not the hang-up lane.Attribution keyed on a
*.drift.tsstack frame, and there is no such frame — every frame isws-providers.tsor a node internal, because the throw is raised from a socket callback the probe's frame does not survive. It is absent, not deeper, so no frame pattern recovers it. The suite title carries the answer, so the frame is tried first (still load-bearing) and the title is the fallback, anchored at the start against the same closedWS_HANDSHAKE_PROBEStable via each surface's registry label.Red reproduces CI byte-identically at
REAL_RC=5. Green givesREAL_RC=2withprovider=Gemini Live,builderFile=src/ws-gemini-live.ts,id=ws-close:1007, and Google's own reason preserved into the finding. Exit 2 is tolerated on the base leg, so the stop clears.Refusing to guess an owner is retained deliberately — a confidently wrong owner fails open. Unregistered suite with no frame still quarantines at 5; a label appearing mid-title still quarantines; a non-refusal code still routes to the hang-up lane. All three probes attribute on both the refusal and handshake lanes, with each probe's real
describetitle bound to its registry label, because a table that works for one provider is how the previously hardcodedopenai-realtimebug survived.An unfailable guard was deleted: a longest-label-first sort that no input could distinguish, since no WS label is a prefix of another. It is replaced by a test pinning that prefix invariant, so the assumption fails loudly if it ever stops holding.
2. The CLI installs signal handlers before announcing readiness
aimock server listening on …was printed beforeprocess.on("SIGTERM")was registered. A signal landing in that window hit Node's default handler, which re-raises — the process diedcode=null, signal=SIGTERMand never ran its shutdown. Anyone sending SIGTERM to aimock in a container could hit it.The pre-fix artifact fails 35/40 attempts idle and 54/60 under load. Post-fix: 0 failures in 100 attempts. Reverting the reorder reds the guard 10/10.
The guard also fails loudly with
run pnpm buildwhendist/cli.jspredatessrc/cli.ts.pnpm testnever builds and these tests exercisedist/, so a stale artifact previously made the guard silently test the wrong binary — which happened during this work and produced a false conclusion.Scope stated honestly: a SIGTERM arriving before readiness is announced is still fatal (20/20). That window is not closed, and the test name and comment say so rather than implying coverage the code does not provide.
3. The Gemini Live probe requests a modality its model serves
Google's reason string blames the modality request, but the cause is one layer back:
fetchLiveCapableTextModelsselected models with!name.includes("native-audio")— a name heuristic standing in for a capability check.gemini-3.1-flash-live-previewis a native-audio model whose name omits that substring, so it was silently misclassified as text-capable and the probe asked it for TEXT.Selection now keys on the declared
bidiGenerateContentcapability. The probe requests["AUDIO"]— native-audio Live models support onlyAUDIO, one modality per session, per Google's capabilities guide, which names this model. The SDK shape becomesinlineData+turnComplete, the drift server gains an audio fixture (the mock already implemented the audio shape), and both legs send identicalgenerationConfig.An
expect(true).toBe(true)canary was deleted rather than kept.Red replays Google's frame locally:
code=1007, 6 failures. Green: 12 pass. Eleven mutations all red, including restoring the name heuristic, flipping the fixture to text, and conflating audio with text in the summary.Verification
Full suite 5280 passing,
test:drift, lint,format:check, build, commitlint all clean. Tests-inclusive typecheck at the 115 baseline with zero errors in every touched file, harness mutation-tested so the zero is a measurement rather than an artifact of a program that compiled nothing. Version-neutral:package.json,CHANGELOG.md,charts/**and.claude-plugin/**untouched.Unproven, and two findings not fixed here
Only a live run proves the Gemini endpoint accepts
["AUDIO"]and emits this sequence; there are no Google credentials in the environment where this was verified. Only a real CI run proves the cron clears.aimockignoresgenerationConfigentirely — removing it from the mock leg fails nothing. It is sent for parity, and that parity is unguarded.src/ws-gemini-live.ts:468-505— the audio branch emitsinlineData+turnCompleteand returns, dropping thetoolCalls/contentcompanions thattypes.ts:398-406documents as preserved "so the tool call / content / reasoning are not silently discarded."