Repository navigation
fix: don't tear down a live WSS subscription on an empty frame (FEMSA-79) - #89
palantir-valiot[bot] wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Overall Assessment
The fix is correct and well-scoped: a zero-length frame is now branched on before orjson.loads, a CLOSE frame (connected is False) still reconnects at WARNING, and a live empty frame is skipped at DEBUG. I checked the websocket-client source: recv() returns "" for a CLOSE frame (after send_close() sets connected = False) and also for a zero-length text frame on a live socket, so the connected discriminator is the right one. No blocking bugs found.
Findings
| # | Severity | File | Issue |
|---|---|---|---|
| 1 | Low | pygqlc/GraphQLClient.py | The live-empty-frame branch continues without restoring settimeout(self.websocket_timeout), so a stream of empty frames keeps the socket on the 0.5 s recv timeout. |
Notes
- Test coverage is good: the three new tests pin the live-empty-frame skip, the close-frame WARNING reconnect, and that non-empty malformed JSON still halts at ERROR. They are hermetic (no sockets,
poll_interval = 0). - The
getattr(self._conn, "connected", True) is Falsedefault is the safe direction: an object withoutconnectedis treated as live, so it skips rather than reconnects. - Considered but not flagged: the
time.sleep(self.poll_interval)in the live-empty branch is redundant with the loop's trailing sleep, but it is harmless and mirrors the existing timeout branch. - Considered but not flagged:
_waiting_connection_ack(line 863) still callsorjson.loadsonrecv()unguarded, so an empty frame during the ack handshake would raise. That is pre-existing code outside this diff and a different failure path; worth a follow-up if the same server ever sends empty frames during handshake. - The
uv.lockrefresh and theruff formatreflow intest_async_client_per_loop.pyare unrelated to the fix but match the repo's per-PR dependency-refresh convention in AGENTS.md.
🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)
|
The PR file list ( None of these files configure Snyk. The check reports an account limit, not a vulnerability in the change. |
There was a problem hiding this comment.
Overall Assessment
The PR stops _sub_routing_loop from parsing a zero-length WSS frame, so keepalive noise no longer logs ERROR and tears the subscription down. The fix is small and correct: the timeout is restored before every branch, a disconnected socket still reconnects (now at WARNING), and a non-empty malformed payload still halts at ERROR. No blocking bugs found.
Findings
No actionable findings.
Notes
- The earlier finding (live-empty-frame branch left the socket on the 0.5 s recv timeout) is fixed:
settimeout(self.websocket_timeout)now runs at line 600, before theif not rawbranch, and the new test asserts the exactsettimeoutcall sequence. getattr(self._conn, "connected", True) is Falsedefaults to "live" when the attribute is missing, so a mock or a future socket wrapper withoutconnectedskips the frame instead of reconnecting. That is the safer default for this bug.- The three new tests cover the live empty frame (
b""and""), the close frame, and non-empty malformed JSON, and they stay hermetic (mock socket,poll_interval = 0). - The
uv.lockrefresh and thetest_async_client_per_loop.pyreformat are unrelated to the fix but match the repo'suv lock --upgradeandruff formatrules. - Not flagged: the PR description says the once-a-minute cadence is assumed to be a live empty frame, not a close. If it is actually a close,
connectedis alreadyFalseand the loop still reconnects, so the fix does not regress that path.
🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)
|
No code change from this review. The timeout restore is already on 4322811, and the reviewer found nothing else to fix. Merge is still blocked for two reasons I cannot clear from this repo:
|
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
The socket timeout is restored toself.websocket_timeoutonly afterrecv()returns. Ifrecv()… · New This assertion is quite brittle because it depends on the internal poll timeout value (0.5) and… · New This changelog bullet is a single very long line, which makes diffs/review harder and reduces… · New Thispatch(...)call is formatted differently from the other updated blocks in the same file… · New
What changed in this PR
Fixes a production issue where empty WSS frames were mis-parsed as JSON, causing the subscription router to treat live sockets as dead and repeatedly reconnect (FEMSA-79).
Changes:
- Update
_sub_routing_loopto skip zero-length frames on connected sockets and treat empty recv on disconnected sockets as a close (WARNING + reconnect path). - Add targeted tests for empty-frame, close-frame, and malformed-JSON behaviors.
- Bump version to 3.10.1 and document the fix in the changelog.
| File | Description |
|---|---|
pygqlc/GraphQLClient.py |
Adjusts WSS routing loop handling for empty frames vs. close frames and restores socket timeouts before parsing. |
tests/pygqlc/gql_client/test_subscribe.py |
Adds regression tests covering FEMSA-79 and related edge cases (empty frame/close/malformed JSON). |
tests/pygqlc/gql_client/test_async_client_per_loop.py |
Reformats patch(...) blocks for readability/consistency. |
pygqlc/__version__.py |
Bumps package version to 3.10.1. |
CHANGELOG.md |
Adds a 3.10.1 entry describing the fix and dependency refresh. |
Review effort: Lite
| try: | ||
| self._conn.settimeout(0.5) | ||
| raw = self._conn.recv() | ||
| # websocket-client returns "" for a CLOSE frame (and has already | ||
| # marked the socket disconnected) and ""/b"" for a zero-length | ||
| # data frame on a live socket. orjson.loads raises JSONDecodeError | ||
| # on either, which is not a transient socket error, so both used | ||
| # to log ERROR and halt. A close must reconnect; a live empty | ||
| # frame must not — that reconnect drops in-flight subscriptions. | ||
| if not raw: | ||
| if getattr(self._conn, "connected", True) is False: | ||
| log( | ||
| LogLevel.WARNING, | ||
| "WSS connection reset or closed by peer", | ||
| ) | ||
| self.wss_conn_halted = True | ||
| else: | ||
| log(LogLevel.DEBUG, "ignoring zero-length WSS frame") | ||
| time.sleep(self.poll_interval) | ||
| continue | ||
| message = orjson.loads(raw) | ||
| self._conn.settimeout(self.websocket_timeout) | ||
| except (TimeoutError, websocket.WebSocketTimeoutException): | ||
| time.sleep(self.poll_interval) | ||
| continue | ||
| except Exception as e: | ||
| if not self.closing: |
| assert not any("Some error trying to receive WSS" in msg for _, msg in records) | ||
| assert not any("invalid WSS message" in msg for _, msg in records) | ||
|
|
||
|
|
||
| def test_sub_routing_loop_close_frame_halts_as_warning(routing_client): | ||
| """websocket-client returns "" for a CLOSE frame after setting connected=False. | ||
| That empty recv is a real close, not keepalive: log WARNING and reconnect, | ||
| do not parse it and do not log ERROR.""" | ||
| gql = routing_client |
|
|
||
| ## [3.10.1] - 2026-11-02 | ||
|
|
||
| - [Fixed] `_sub_routing_loop` no longer parses a zero-length WSS frame. `orjson.loads` on an empty payload raises `JSONDecodeError`, which is not in `TRANSIENT_WS_ERRORS`, so the generic handler logged ERROR and set `wss_conn_halted`. `websocket-client` returns `""` for a CLOSE frame (socket already disconnected) and `""`/`b""` for an empty data frame on a live socket. A disconnected socket still reconnects, at WARNING. A live empty frame is skipped at DEBUG, so keepalive noise no longer tears the subscription down. A non-empty payload that is not JSON still halts at ERROR. (FEMSA-79) |
|
|
||
| with patch("pygqlc.GraphQLClient.httpx.AsyncClient", side_effect=make_client) as ctor: | ||
| with patch( | ||
| "pygqlc.GraphQLClient.httpx.AsyncClient", side_effect=make_client |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
5 open issues
The socket timeout is restored toself.websocket_timeoutonly on the success path (afterrecv()… · New The socket timeout is restored toself.websocket_timeoutonly afterrecv()returns. Ifrecv()… This assertion is quite brittle because it depends on the internal poll timeout value (0.5) and… Thispatch(...)call is formatted differently from the other updated blocks in the same file… This changelog bullet is a single very long line, which makes diffs/review harder and reduces…
Review effort: Lite
| try: | ||
| self._conn.settimeout(0.5) | ||
| raw = self._conn.recv() | ||
| message = orjson.loads(raw) | ||
| # Restore before any branch. The ping thread shares this socket, | ||
| # and a stream of empty frames must not leave it on the 0.5s poll. | ||
| self._conn.settimeout(self.websocket_timeout) | ||
| # websocket-client returns "" for a CLOSE frame (and has already | ||
| # marked the socket disconnected) and ""/b"" for a zero-length | ||
| # data frame on a live socket. orjson.loads raises JSONDecodeError | ||
| # on either, which is not a transient socket error, so both used | ||
| # to log ERROR and halt. A close must reconnect; a live empty | ||
| # frame must not — that reconnect drops in-flight subscriptions. | ||
| if not raw: | ||
| if getattr(self._conn, "connected", True) is False: | ||
| log( | ||
| LogLevel.WARNING, | ||
| "WSS connection reset or closed by peer", | ||
| ) | ||
| self.wss_conn_halted = True | ||
| else: | ||
| log(LogLevel.DEBUG, "ignoring zero-length WSS frame") | ||
| time.sleep(self.poll_interval) | ||
| continue | ||
| message = orjson.loads(raw) | ||
| except (TimeoutError, websocket.WebSocketTimeoutException): | ||
| time.sleep(self.poll_interval) | ||
| continue |
orjson.loads on an empty frame raises JSONDecodeError, which is not a transient socket error, so the router logged ERROR and tore the connection down. Skip the frame instead.
websocket-client returns "" for a CLOSE frame after marking the socket disconnected, and ""/b"" for a live empty data frame. Skip the live frame; treat a disconnected socket as a peer close.
The router polls recv at 0.5s, then puts the shared socket back on websocket_timeout. Skipping a live empty frame used to continue before that restore, so a stream of them left ping sends on the short timeout.
|
Rebased onto current
|
⏸️ Awaiting human inputThe rebase onto main is done locally at 6c2aac7 (version 3.10.2). git push-safe exited 3 because its recovery replayed FEMSA-80 onto the old PR tip and conflicted on CHANGELOG.md. May I force-with-lease push 6c2aac7 to palantir/FEMSA-79-skip-empty-wss-frames? Reply on this PR or continue the agent chat: 🤖 Implementer · Grok 4.7 (xAI) |
4322811 to
6c2aac7
Compare
|
Rebased onto current Version is 3.10.2 so it does not collide with the 3.10.1 FEMSA-80 release. FEMSA-80's |
There was a problem hiding this comment.
Overall Assessment
The fix is correct and well scoped: a zero-length frame is no longer handed to orjson.loads, a disconnected socket still reconnects (now at WARNING), and a live empty frame is skipped without tearing down the subscription. The timeout restore from the earlier review is in place and pinned by a test. No blocking bugs found.
Findings
No actionable findings.
Notes
- Fixed since last review: the live-empty-frame branch no longer leaves the shared socket on the 0.5 s poll timeout.
settimeout(self.websocket_timeout)now runs immediately afterrecvreturns (GraphQLClient.py:614), andtest_sub_routing_loop_empty_frame_does_not_haltasserts the0.5, 60, 0.5, 60, 0.5, 60sequence. - The
getattr(self._conn, "connected", True) is Falseguard is the right default: a mock or a future connection object withoutconnectedis treated as live, so the loop skips rather than reconnects. I checked websocket-client'srecv_data_frame: onOPCODE_CLOSEit callssend_close(), which setsself.connected = Falsebeforerecvreturns"", so the close path is reachable as described.pyproject.tomlpinswebsocket-client>=1.8, and that behavior is present in the versions in range. - Considered but not flagged: a server that streams empty frames continuously makes the loop spin at
poll_interval(5 ms) with a DEBUG log per frame. That is the same cost as the existingpong/connection_ackno-op branches, so it is not a new hot path. - Test coverage is good: the three new tests cover live empty frame, close frame, and malformed non-empty payload, and each asserts on log levels, so a regression to ERROR would fail. The
test_async_client_per_loop.pychanges areruff formatonly. - Version bump and CHANGELOG entry follow the repo's release rules (patch bump in
__version__.py,## [x.y.z] - YYYY-MM-DDentry with a[Fixed]line).
🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)
|
No code change from this review. The timeout restore and the empty-frame split are already on Merge is still blocked by |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
8 open issues
The socket timeout is restored toself.websocket_timeoutonly on the success path (afterrecv()… The socket timeout is restored toself.websocket_timeoutonly afterrecv()returns. Ifrecv()… The socket timeout restoration only happens after a successfulrecv(). Ifrecv()raises (e.g.,… · New In the CLOSE-frame path (connected is False), the loop immediatelycontinues without any… · New This assertion is quite brittle because it depends on the internal poll timeout value (0.5) and… This assertion hard-codes the fullsettimeoutcall sequence, which makes the test brittle to… · New Thispatch(...)call is formatted differently from the other updated blocks in the same file… This changelog bullet is a single very long line, which makes diffs/review harder and reduces…
Review effort: Lite
| try: | ||
| self._conn.settimeout(0.5) | ||
| raw = self._conn.recv() | ||
| message = orjson.loads(raw) | ||
| # Restore before any branch. The ping thread shares this socket, | ||
| # and a stream of empty frames must not leave it on the 0.5s poll. | ||
| self._conn.settimeout(self.websocket_timeout) |
| if not raw: | ||
| if getattr(self._conn, "connected", True) is False: | ||
| log( | ||
| LogLevel.WARNING, | ||
| "WSS connection reset or closed by peer", | ||
| ) | ||
| self.wss_conn_halted = True | ||
| else: | ||
| log(LogLevel.DEBUG, "ignoring zero-length WSS frame") | ||
| time.sleep(self.poll_interval) | ||
| continue |
| assert [c.args[0] for c in gql._conn.settimeout.call_args_list] == [ | ||
| 0.5, | ||
| gql.websocket_timeout, | ||
| 0.5, | ||
| gql.websocket_timeout, | ||
| 0.5, | ||
| gql.websocket_timeout, | ||
| ] |



Summary
A zero-length WSS frame was parsed as JSON, then treated as a dead connection.
femsa-worker-monitorstore the socket down and resubscribed about once a minute (56 times in an hour). Each window can drop monitor events. The pod stays healthy, so the loss is silent.websocket-clientreturns""for two different things. A CLOSE frame, after it has already marked the socket disconnected. And an empty data frame on a socket that is still connected. Both used to hitorjson.loadsand raiseJSONDecodeError, which is not a transient socket error, so the router logged ERROR and halted.A disconnected socket still reconnects, now at WARNING. A live empty frame is skipped. A non-empty payload that is not JSON still halts at ERROR. Earlier attempts (pygqlc#66, pygqlc#67) either reconnected on every empty recv or treated every
JSONDecodeErroras transient. The first keeps the reconnect storm if the frame is keepalive. The second hides a genuinely malformed payload. I assumed the once-a-minute cadence is a live empty frame, not a close. If it is a close,connectedis already false and we still reconnect.Why
FEMSA-79. Production error in
femsa-worker-monitors(tenant femsa, prod):Test plan
b""and""on a connected socket called_new_conntwice and would have resubscribed.connected is False, recv returns"") still reconnects, at WARNING, with no ERROR.uv run pyteston the hermetic suite: 95 passed. The 25gqlintegration tests needAPI/WSS/TOKENand exit in setup without them; they are unchanged.uvx ruff format --checkanduvx bandit -r . -s B101 -llclean.uv lock --upgrade;uv tree --depth 1 --outdatedreports nothing.Closes FEMSA-79
💬 Continue the agent chat (live transcript, tools, follow-ups):
https://palantir.valiot.io/agent-tasks?task=b131d31a-f572-4754-bd6c-4e60fbdb8530