Skip to content

fix: don't tear down a live WSS subscription on an empty frame (FEMSA-79) - #89

Open
palantir-valiot[bot] wants to merge 3 commits into
mainfrom
palantir/FEMSA-79-skip-empty-wss-frames
Open

palantir-valiot[bot] wants to merge 3 commits into
mainfrom
palantir/FEMSA-79-skip-empty-wss-frames

Conversation

@palantir-valiot

Copy link
Copy Markdown
Contributor

Summary

A zero-length WSS frame was parsed as JSON, then treated as a dead connection. femsa-worker-monitors tore the socket down and resubscribed about once a minute (56 times in an hour). Each window can drop monitor events. The pod stays healthy, so the loss is silent.

websocket-client returns "" for two different things. A CLOSE frame, after it has already marked the socket disconnected. And an empty data frame on a socket that is still connected. Both used to hit orjson.loads and raise JSONDecodeError, which is not a transient socket error, so the router logged ERROR and halted.

A disconnected socket still reconnects, now at WARNING. A live empty frame is skipped. A non-empty payload that is not JSON still halts at ERROR. Earlier attempts (pygqlc#66, pygqlc#67) either reconnected on every empty recv or treated every JSONDecodeError as transient. The first keeps the reconnect storm if the frame is keepalive. The second hides a genuinely malformed payload. I assumed the once-a-minute cadence is a live empty frame, not a close. If it is a close, connected is already false and we still reconnect.

Why

FEMSA-79. Production error in femsa-worker-monitors (tenant femsa, prod):

orjson.JSONDecodeError: Input is a zero-length, empty document: line 1 column 1 (char 0)
  File ".../pygqlc/GraphQLClient.py", line 598, in _sub_routing_loop
    message = orjson.loads(raw)

Test plan

  • Failing test first: an empty b"" and "" on a connected socket called _new_conn twice and would have resubscribed.
  • Close frame (connected is False, recv returns "") still reconnects, at WARNING, with no ERROR.
  • Non-empty malformed JSON still logs ERROR and halts.
  • uv run pytest on the hermetic suite: 95 passed. The 25 gql integration tests need API/WSS/TOKEN and exit in setup without them; they are unchanged.
  • uvx ruff format --check and uvx bandit -r . -s B101 -ll clean.
  • uv lock --upgrade; uv tree --depth 1 --outdated reports nothing.

Closes FEMSA-79


💬 Continue the agent chat (live transcript, tools, follow-ups):
https://palantir.valiot.io/agent-tasks?task=b131d31a-f572-4754-bd6c-4e60fbdb8530

@palantir-valiot
palantir-valiot Bot requested review from acrogenesis, barucAlmaguer and doctorcorral and a balanced review from Copilot November 2, 2026 17:34

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

The fix is correct and well-scoped: a zero-length frame is now branched on before orjson.loads, a CLOSE frame (connected is False) still reconnects at WARNING, and a live empty frame is skipped at DEBUG. I checked the websocket-client source: recv() returns "" for a CLOSE frame (after send_close() sets connected = False) and also for a zero-length text frame on a live socket, so the connected discriminator is the right one. No blocking bugs found.

Findings

# Severity File Issue
1 Low pygqlc/GraphQLClient.py The live-empty-frame branch continues without restoring settimeout(self.websocket_timeout), so a stream of empty frames keeps the socket on the 0.5 s recv timeout.

Notes

  • Test coverage is good: the three new tests pin the live-empty-frame skip, the close-frame WARNING reconnect, and that non-empty malformed JSON still halts at ERROR. They are hermetic (no sockets, poll_interval = 0).
  • The getattr(self._conn, "connected", True) is False default is the safe direction: an object without connected is treated as live, so it skips rather than reconnects.
  • Considered but not flagged: the time.sleep(self.poll_interval) in the live-empty branch is redundant with the loop's trailing sleep, but it is harmless and mirrors the existing timeout branch.
  • Considered but not flagged: _waiting_connection_ack (line 863) still calls orjson.loads on recv() unguarded, so an empty frame during the ack handshake would raise. That is pre-existing code outside this diff and a different failure path; worth a follow-up if the same server ever sends empty frames during handshake.
  • The uv.lock refresh and the ruff format reflow in test_async_client_per_loop.py are unrelated to the fix but match the repo's per-PR dependency-refresh convention in AGENTS.md.

🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)

Comment thread pygqlc/GraphQLClient.py Resolved
@palantir-valiot

Copy link
Copy Markdown
Contributor Author

The security/snyk (valiot) check is failing on a quota, not on this diff:

security/snyk (valiot)	fail	You have used your limit of private tests
https://app.snyk.io/org/valiot/pr-checks/87a4cf40-7eb1-4b5e-8b66-8f93bf0deb87

PR file list (gh pr view 89 --json files --jq '.files[].path'):

CHANGELOG.md
pygqlc/GraphQLClient.py
pygqlc/__version__.py
tests/pygqlc/gql_client/test_async_client_per_loop.py
tests/pygqlc/gql_client/test_subscribe.py
uv.lock

None of these files configure Snyk. The check reports an account limit, not a vulnerability in the change. tests and check_outdated passed.

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

The PR stops _sub_routing_loop from parsing a zero-length WSS frame, so keepalive noise no longer logs ERROR and tears the subscription down. The fix is small and correct: the timeout is restored before every branch, a disconnected socket still reconnects (now at WARNING), and a non-empty malformed payload still halts at ERROR. No blocking bugs found.

Findings

No actionable findings.

Notes

  • The earlier finding (live-empty-frame branch left the socket on the 0.5 s recv timeout) is fixed: settimeout(self.websocket_timeout) now runs at line 600, before the if not raw branch, and the new test asserts the exact settimeout call sequence.
  • getattr(self._conn, "connected", True) is False defaults to "live" when the attribute is missing, so a mock or a future socket wrapper without connected skips the frame instead of reconnecting. That is the safer default for this bug.
  • The three new tests cover the live empty frame (b"" and ""), the close frame, and non-empty malformed JSON, and they stay hermetic (mock socket, poll_interval = 0).
  • The uv.lock refresh and the test_async_client_per_loop.py reformat are unrelated to the fix but match the repo's uv lock --upgrade and ruff format rules.
  • Not flagged: the PR description says the once-a-minute cadence is assumed to be a live empty frame, not a close. If it is actually a close, connected is already False and the loop still reconnects, so the fix does not regress that path.

🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

No code change from this review. The timeout restore is already on 4322811, and the reviewer found nothing else to fix.

Merge is still blocked for two reasons I cannot clear from this repo:

  • reviewDecision is REVIEW_REQUIRED. The reviewer left a comment, not an approval.
  • security/snyk (valiot) is still error: You have used your limit of private tests. That is an account quota, not a finding in this diff. tests and check_outdated passed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues
What changed in this PR

Fixes a production issue where empty WSS frames were mis-parsed as JSON, causing the subscription router to treat live sockets as dead and repeatedly reconnect (FEMSA-79).

Changes:

  • Update _sub_routing_loop to skip zero-length frames on connected sockets and treat empty recv on disconnected sockets as a close (WARNING + reconnect path).
  • Add targeted tests for empty-frame, close-frame, and malformed-JSON behaviors.
  • Bump version to 3.10.1 and document the fix in the changelog.
File Description
pygqlc/​GraphQLClient.py Adjusts WSS routing loop handling for empty frames vs. close frames and restores socket timeouts before parsing.
tests/​pygqlc/​gql_client/​test_subscribe.py Adds regression tests covering FEMSA-79 and related edge cases (empty frame/close/malformed JSON).
tests/​pygqlc/​gql_client/​test_async_client_per_loop.py Reformats patch(...) blocks for readability/consistency.
pygqlc/​__version__.py Bumps package version to 3.10.1.
CHANGELOG.md Adds a 3.10.1 entry describing the fix and dependency refresh.

Review effort: Lite

Comment thread pygqlc/GraphQLClient.py
Comment on lines 595 to 621
try:
self._conn.settimeout(0.5)
raw = self._conn.recv()
# websocket-client returns "" for a CLOSE frame (and has already
# marked the socket disconnected) and ""/b"" for a zero-length
# data frame on a live socket. orjson.loads raises JSONDecodeError
# on either, which is not a transient socket error, so both used
# to log ERROR and halt. A close must reconnect; a live empty
# frame must not — that reconnect drops in-flight subscriptions.
if not raw:
if getattr(self._conn, "connected", True) is False:
log(
LogLevel.WARNING,
"WSS connection reset or closed by peer",
)
self.wss_conn_halted = True
else:
log(LogLevel.DEBUG, "ignoring zero-length WSS frame")
time.sleep(self.poll_interval)
continue
message = orjson.loads(raw)
self._conn.settimeout(self.websocket_timeout)
except (TimeoutError, websocket.WebSocketTimeoutException):
time.sleep(self.poll_interval)
continue
except Exception as e:
if not self.closing:
Comment on lines +269 to +277
assert not any("Some error trying to receive WSS" in msg for _, msg in records)
assert not any("invalid WSS message" in msg for _, msg in records)


def test_sub_routing_loop_close_frame_halts_as_warning(routing_client):
"""websocket-client returns "" for a CLOSE frame after setting connected=False.
That empty recv is a real close, not keepalive: log WARNING and reconnect,
do not parse it and do not log ERROR."""
gql = routing_client
Comment thread CHANGELOG.md Outdated

## [3.10.1] - 2026-11-02

- [Fixed] `_sub_routing_loop` no longer parses a zero-length WSS frame. `orjson.loads` on an empty payload raises `JSONDecodeError`, which is not in `TRANSIENT_WS_ERRORS`, so the generic handler logged ERROR and set `wss_conn_halted`. `websocket-client` returns `""` for a CLOSE frame (socket already disconnected) and `""`/`b""` for an empty data frame on a live socket. A disconnected socket still reconnects, at WARNING. A live empty frame is skipped at DEBUG, so keepalive noise no longer tears the subscription down. A non-empty payload that is not JSON still halts at ERROR. (FEMSA-79)

with patch("pygqlc.GraphQLClient.httpx.AsyncClient", side_effect=make_client) as ctor:
with patch(
"pygqlc.GraphQLClient.httpx.AsyncClient", side_effect=make_client
Copilot AI balanced review requested due to automatic review settings November 2, 2026 17:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues

Review effort: Lite

Comment thread pygqlc/GraphQLClient.py
Comment on lines 595 to 621
try:
self._conn.settimeout(0.5)
raw = self._conn.recv()
message = orjson.loads(raw)
# Restore before any branch. The ping thread shares this socket,
# and a stream of empty frames must not leave it on the 0.5s poll.
self._conn.settimeout(self.websocket_timeout)
# websocket-client returns "" for a CLOSE frame (and has already
# marked the socket disconnected) and ""/b"" for a zero-length
# data frame on a live socket. orjson.loads raises JSONDecodeError
# on either, which is not a transient socket error, so both used
# to log ERROR and halt. A close must reconnect; a live empty
# frame must not — that reconnect drops in-flight subscriptions.
if not raw:
if getattr(self._conn, "connected", True) is False:
log(
LogLevel.WARNING,
"WSS connection reset or closed by peer",
)
self.wss_conn_halted = True
else:
log(LogLevel.DEBUG, "ignoring zero-length WSS frame")
time.sleep(self.poll_interval)
continue
message = orjson.loads(raw)
except (TimeoutError, websocket.WebSocketTimeoutException):
time.sleep(self.poll_interval)
continue
orjson.loads on an empty frame raises JSONDecodeError, which is not a
transient socket error, so the router logged ERROR and tore the
connection down. Skip the frame instead.
websocket-client returns "" for a CLOSE frame after marking the socket
disconnected, and ""/b"" for a live empty data frame. Skip the live
frame; treat a disconnected socket as a peer close.
The router polls recv at 0.5s, then puts the shared socket back on
websocket_timeout. Skipping a live empty frame used to continue before
that restore, so a stream of them left ping sends on the short timeout.
@palantir-valiot

Copy link
Copy Markdown
Contributor Author

Rebased onto current main locally (6c2aac7). The empty-frame bug is not on main — FEMSA-80 only treats a dead-socket OSError as transient. Version is now 3.10.2 because 3.10.1 is the FEMSA-80 release.

git push-safe exited 3. Its recovery tried to replay FEMSA-80 onto the old PR tip and conflicted on CHANGELOG.md. I did not force-push. The local branch is the rebased history; the remote branch is still the pre-rebase commits.

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

⏸️ Awaiting human input

The rebase onto main is done locally at 6c2aac7 (version 3.10.2). git push-safe exited 3 because its recovery replayed FEMSA-80 onto the old PR tip and conflicted on CHANGELOG.md. May I force-with-lease push 6c2aac7 to palantir/FEMSA-79-skip-empty-wss-frames?
Reason: rebase conflict on CHANGELOG.md

Reply on this PR or continue the agent chat:
https://palantir.valiot.io/agent-tasks?task=b131d31a-f572-4754-bd6c-4e60fbdb8530


🤖 Implementer · Grok 4.7 (xAI)

@palantir-valiot
palantir-valiot Bot force-pushed the palantir/FEMSA-79-skip-empty-wss-frames branch from 4322811 to 6c2aac7 Compare November 2, 2026 22:31
Copilot AI balanced review requested due to automatic review settings November 2, 2026 22:31
@palantir-valiot

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and pushed 6c2aac7. The empty-frame fix is still needed: main only added the FEMSA-80 dead-socket OSError path, and _sub_routing_loop still calls orjson.loads on a zero-length frame.

Version is 3.10.2 so it does not collide with the 3.10.1 FEMSA-80 release. FEMSA-80's _is_transient_ws_error check is kept.

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

The fix is correct and well scoped: a zero-length frame is no longer handed to orjson.loads, a disconnected socket still reconnects (now at WARNING), and a live empty frame is skipped without tearing down the subscription. The timeout restore from the earlier review is in place and pinned by a test. No blocking bugs found.

Findings

No actionable findings.

Notes

  • Fixed since last review: the live-empty-frame branch no longer leaves the shared socket on the 0.5 s poll timeout. settimeout(self.websocket_timeout) now runs immediately after recv returns (GraphQLClient.py:614), and test_sub_routing_loop_empty_frame_does_not_halt asserts the 0.5, 60, 0.5, 60, 0.5, 60 sequence.
  • The getattr(self._conn, "connected", True) is False guard is the right default: a mock or a future connection object without connected is treated as live, so the loop skips rather than reconnects. I checked websocket-client's recv_data_frame: on OPCODE_CLOSE it calls send_close(), which sets self.connected = False before recv returns "", so the close path is reachable as described. pyproject.toml pins websocket-client>=1.8, and that behavior is present in the versions in range.
  • Considered but not flagged: a server that streams empty frames continuously makes the loop spin at poll_interval (5 ms) with a DEBUG log per frame. That is the same cost as the existing pong/connection_ack no-op branches, so it is not a new hot path.
  • Test coverage is good: the three new tests cover live empty frame, close frame, and malformed non-empty payload, and each asserts on log levels, so a regression to ERROR would fail. The test_async_client_per_loop.py changes are ruff format only.
  • Version bump and CHANGELOG entry follow the repo's release rules (patch bump in __version__.py, ## [x.y.z] - YYYY-MM-DD entry with a [Fixed] line).

🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

No code change from this review. The timeout restore and the empty-frame split are already on 6c2aac7, and the reviewer found nothing else to fix.

Merge is still blocked by security/snyk (valiot): You have used your limit of private tests. That is an account quota, not a finding in this diff. tests and check_outdated passed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread pygqlc/GraphQLClient.py
Comment on lines 609 to 614
try:
self._conn.settimeout(0.5)
raw = self._conn.recv()
message = orjson.loads(raw)
# Restore before any branch. The ping thread shares this socket,
# and a stream of empty frames must not leave it on the 0.5s poll.
self._conn.settimeout(self.websocket_timeout)
Comment thread pygqlc/GraphQLClient.py
Comment on lines +621 to +631
if not raw:
if getattr(self._conn, "connected", True) is False:
log(
LogLevel.WARNING,
"WSS connection reset or closed by peer",
)
self.wss_conn_halted = True
else:
log(LogLevel.DEBUG, "ignoring zero-length WSS frame")
time.sleep(self.poll_interval)
continue
Comment on lines +325 to +332
assert [c.args[0] for c in gql._conn.settimeout.call_args_list] == [
0.5,
gql.websocket_timeout,
0.5,
gql.websocket_timeout,
0.5,
gql.websocket_timeout,
]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant