Skip to content

fix: skip a live empty WSS frame instead of resubscribing (LAMGTO-59) - #91

Open
palantir-valiot[bot] wants to merge 1 commit into
mainfrom
palantir/LAMGTO-59-skip-empty-wss-frames
Open

palantir-valiot[bot] wants to merge 1 commit into
mainfrom
palantir/LAMGTO-59-skip-empty-wss-frames

Conversation

@palantir-valiot

Copy link
Copy Markdown
Contributor

Summary

A zero-length WSS frame was parsed as JSON, then treated as a dead connection. lamosa-gto-services-workflows logged that at ERROR and resubscribed every subscription. The pod stayed healthy. The loss is the false alarm and the dropped subscription window.

websocket-client.recv() returns "" for two different things. A CLOSE frame, after it has already marked the socket disconnected. And an empty data frame on a socket that is still connected. Both hit orjson.loads and raise JSONDecodeError, which is not a transient socket error, so the router logged ERROR and halted.

A disconnected socket still reconnects, now at WARNING. A live empty frame is skipped. A non-empty payload that is not JSON still halts at ERROR. I assumed the observed empty document is that recv() return, not a server sending the JSON value "". If it is a real close, connected is already false and we still reconnect. The change is in pygqlc/GraphQLClient.py in _sub_routing_loop.

pygqlc#89 (FEMSA-79) is the same split and a reviewer already accepted it, but it never merged. This lands the fix on current main. If #89 merges first, both claim 3.10.2 — close one of them.

Why

LAMGTO-59. Production error in lamosa-gto-services-workflows (tenant lamosa-gto, prod), pod lamosa-gto-services-workflows-77869dff86-pc7rs:

orjson.JSONDecodeError: Input is a zero-length, empty document: line 1 column 1 (char 0)
  File ".../pygqlc/GraphQLClient.py", line 612, in _sub_routing_loop
    message = orjson.loads(raw)

Upstream socket churn on the identity websocket is not ours. Classifying that empty recv as a payload bug, and tearing every subscription down for it, is.

Test plan

  • Failing test first: an empty b"" and "" on a connected socket called _new_conn twice and would have resubscribed.
  • Close frame (connected is False, recv returns "") still reconnects, at WARNING, with no ERROR.
  • Non-empty malformed JSON still logs ERROR and halts.
  • uv run pytest on the hermetic suite: 100 passed. The integration tests that need API/WSS/TOKEN were not run here; they are unchanged.
  • uvx ruff format --check and uvx bandit -r . -s B101 -ll clean. ruff format also reflowed test_async_client_per_loop.py; no behavior change.
  • uv lock --upgrade; uv tree --depth 1 --outdated reports nothing.
  • Bug fix, no API change. No UI, so no screenshots.

Closes LAMGTO-59


💬 Continue the agent chat (live transcript, tools, follow-ups):
https://palantir.valiot.io/agent-tasks?task=37d53c4a-7c19-4c30-8be8-6b55678e2a78

websocket-client.recv() returns "" for a CLOSE frame after marking the
socket disconnected, and ""/b"" for a zero-length data frame on a live
socket. orjson.loads on that raises JSONDecodeError, which is not a
transient socket error, so the router logged ERROR and tore every
subscription down. Skip the live frame. A real close still reconnects,
at WARNING.
@palantir-valiot

Copy link
Copy Markdown
Contributor Author

The tests check is failing because the live integration API returned 502, not because of this diff.

The first error, from gh run view 44960260387 --log-failed:

ERROR tests/pygqlc/gql_client/test_async.py::test_async_query_no_errors - SystemExit: [{'message': 'Query failed to run by returning code of 502.
{"title":"Error 502: Bad gateway","status":502,"zone":"test.valiot.app","cloudflare_error":true,"ray_id":"a5e35b0319c0e102","timestamp":"2026-11-21T23:08:17Z"}
query {authors{id}}'}]
tests/conftest.py:63: SystemExit

tests/conftest.py calls sys.exit when that authors query returns errors. The session fixture then dies, and the other 24 integration tests error in pytest setup (assert not self._finalizers). The hermetic suite passed: 100 passed, 25 errors.

A probe of https://test.valiot.app/ just now still returns 502, so a rerun would fail the same way. Main's last PygqlClient [CI] run (2026-11-02, run 41530198422) passed while that origin was up.

PR file list (gh pr view 91 --json files --jq '.files[].path'):

CHANGELOG.md
pygqlc/GraphQLClient.py
pygqlc/__version__.py
tests/pygqlc/gql_client/test_async_client_per_loop.py
tests/pygqlc/gql_client/test_subscribe.py
uv.lock

None of these files is the authors query, tests/conftest.py, or the Cloudflare origin. The new tests are the hermetic routing-loop cases, and those 100 passed.

security/snyk (valiot) is also failing on a quota, not on this diff:

security/snyk (valiot)	fail	You have used your limit of private tests
https://app.snyk.io/org/valiot/pr-checks/1f7e609d-e06d-4ae7-91c7-cc3b50b3dc4b

Same file list. Nothing here configures Snyk. check_outdated and CodeQL passed.

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

The PR splits the two meanings of an empty recv() in _sub_routing_loop: a CLOSE frame (connected already False) reconnects at WARNING, and a zero-length data frame on a live socket is skipped at DEBUG instead of raising JSONDecodeError and tearing down every subscription. The discriminator is sound — websocket-client's send_close() sets connected = False before recv() returns the close frame — and the timeout restore was correctly moved ahead of the branch so a stream of empty frames cannot leave the shared socket on the 0.5s poll. No blocking bugs found.

Findings

No actionable findings.

Notes

  • Test coverage is good: three new tests pin the split (live empty frame skipped with no _new_conn, close frame reconnects at WARNING, non-empty malformed JSON still ERROR), and the empty-frame test also asserts the exact settimeout sequence so the timeout-restore regression is caught. All three fail against the pre-fix code for the right reason.
  • Considered but not flagged: _waiting_connection_ack (GraphQLClient.py:880) still does orjson.loads(self._conn.recv()) with no empty-frame guard. It is the same class of bug on the connect path, but the failure mode is a failed connect (_new_conn logs ERROR and returns False, the router retries with backoff), not a subscription teardown, and the PR scopes the change to _sub_routing_loop.
  • Considered but not flagged: a persistent stream of empty frames now spins the router at ~200 iterations/s (poll_interval 0.005) and never reconnects. The author explicitly chose to skip live empty frames, and the ping thread still detects a genuinely dead socket, so this is a design trade-off rather than a defect.
  • getattr(self._conn, "connected", True) is False treats a missing attribute as live. websocket-client always sets connected, so this is defensive only; the is False comparison also avoids treating a falsy non-bool as disconnected.
  • The [Dependencies] CHANGELOG line and the uv.lock refresh are consistent with the lock contents I could read (certifi 2026.11.20, requires-python = ">=3.11" matching pyproject.toml).

🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

No code change from this review. The empty-frame split, the timeout restore, and the three regression tests are already on 6d47981, and the reviewer found nothing else to fix.

_waiting_connection_ack is the same class of empty recv(), but a failed handshake already retries through _new_conn. I left it out of this PR on purpose.

Merge is still blocked for two reasons I cannot clear from this repo:

  • tests failed because test.valiot.app returned Cloudflare 502 during the live gql fixture. The hermetic suite passed (100). The origin was still 502 when I probed it.
  • security/snyk (valiot) is You have used your limit of private tests. That is an account quota, not a finding in this diff.

check_outdated and CodeQL passed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues
What changed in this PR

Fixes false WSS reconnect/resubscribe storms by treating a zero-length recv() on a still-connected socket as a skip (DEBUG) rather than a JSON parsing failure (ERROR), while still reconnecting on real CLOSE frames.

Changes:

  • Update _sub_routing_loop to restore socket timeout and skip live empty frames; log WARNING + reconnect on peer close.
  • Add regression tests covering empty data frame vs CLOSE frame vs malformed JSON behavior.
  • Bump version to 3.10.2 and document the fix in the changelog.
File Description
pygqlc/​GraphQLClient.py Adjusts subscription routing loop to distinguish live empty frames from CLOSE and avoid unnecessary resubscription.
tests/​pygqlc/​gql_client/​test_subscribe.py Adds targeted tests for empty frame handling, peer close, and malformed JSON behavior.
tests/​pygqlc/​gql_client/​test_async_client_per_loop.py Formatting-only changes (patch calls reflowed).
pygqlc/​__version__.py Bumps package version to 3.10.2.
CHANGELOG.md Documents the WSS empty-frame fix and dependency refresh notes.

Review effort: Lite

Comment thread pygqlc/GraphQLClient.py
Comment on lines 609 to 614
try:
self._conn.settimeout(0.5)
raw = self._conn.recv()
message = orjson.loads(raw)
# Restore before any branch. The ping thread shares this socket,
# and a stream of empty frames must not leave it on the 0.5s poll.
self._conn.settimeout(self.websocket_timeout)
Comment thread pygqlc/GraphQLClient.py
Comment on lines 634 to 636
except (TimeoutError, websocket.WebSocketTimeoutException):
time.sleep(self.poll_interval)
continue
Comment on lines +327 to +334
assert [c.args[0] for c in gql._conn.settimeout.call_args_list] == [
0.5,
gql.websocket_timeout,
0.5,
gql.websocket_timeout,
0.5,
gql.websocket_timeout,
]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant