Skip to content

fix: treat dead-socket OSError as a transient WSS reconnect (FEMSA-80) - #90

Merged
barucAlmaguer merged 1 commit into
mainfrom
palantir/FEMSA-80-transient-wss-oserror
Nov 2, 2026
Merged

barucAlmaguer merged 1 commit into
mainfrom
palantir/FEMSA-80-transient-wss-oserror

Conversation

@palantir-valiot

Copy link
Copy Markdown
Contributor

Summary

A normal peer close of a WSS subscription was logged as ERROR with a full traceback. The socket still reconnects. The damage is log noise that can hide a real WSS failure.

websocket-client answers that close with a close frame on the already-dead socket. sock.send then raises OSError: [Errno 9] Bad file descriptor. Python does not promote EBADF or ENOTCONN to a Connection* subclass, so the error missed TRANSIENT_WS_ERRORS and took the ERROR branch in _sub_routing_loop.

Why

FEMSA-80, seen once on femsa-worker-monitors in femsa prod. No user-visible loss: wss_conn_halted was already set and the loop reconnects with backoff.

I did not add all of OSError to TRANSIENT_WS_ERRORS. That would also swallow PermissionError and other real bugs. A helper treats EBADF, ENOTCONN, and a bare ECONNRESET as transient. Every other OSError still logs at ERROR. OSError(ECONNRESET) is already a ConnectionResetError when Python constructs it; the errno covers a bare OSError whose errno was copied or assigned later. The ping send path uses the same helper, since it shares the classification.

Reconnect behavior is unchanged. Only the log level moves from ERROR to WARNING for these errnos. Version 3.10.1.

Test plan

  • Wrote the failing tests first. OSError(EBADF) and OSError(ENOTCONN) from recv logged ERROR before the fix, WARNING after.
  • OSError(EACCES) still logs ERROR, so the change does not blanket-catch OSError.
  • A ping send that raises OSError(EBADF) logs WARNING and sets wss_conn_halted.
  • uv run pytest: 97 passed. The 25 live tests that need API / WSS / TOKEN error locally without those secrets. That is pre-existing; CI injects the secrets.
  • uvx ruff format --check
  • uvx bandit -r . -s B101 -ll clean
  • uv lock --upgrade; uv tree --depth 1 --outdated reports nothing

Screenshots not applicable. This is a library change with no UI.

Closes FEMSA-80


💬 Continue the agent chat (live transcript, tools, follow-ups):
https://palantir.valiot.io/agent-tasks?task=b74c4655-8469-4a2c-ada7-dbfccd912693

A peer close makes websocket-client reply with a close frame on an
already-dead socket. That raises OSError(EBADF), which missed
TRANSIENT_WS_ERRORS and was logged as ERROR with a traceback.

Classify EBADF, ENOTCONN, and a bare ECONNRESET as a calm reconnect
on both recv and ping. Other OSErrors still log at ERROR.
@palantir-valiot
palantir-valiot Bot requested review from acrogenesis, barucAlmaguer and doctorcorral and a balanced review from Copilot November 2, 2026 18:10

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

The PR adds _is_transient_ws_error, which classifies a bare OSError with EBADF, ENOTCONN, or ECONNRESET as a transient WSS teardown, and uses it in both _sub_routing_loop and _ping_pong. Reconnect behavior is unchanged; only the log level moves from ERROR to WARNING for those errnos. The implementation is small, matches the existing TRANSIENT_WS_ERRORS pattern, and the tests cover the positive cases plus an EACCES negative control. No blocking bugs found.

Findings

No actionable findings.

Notes

  • Test coverage is good: EBADF/ENOTCONN are parametrized on the recv path, EACCES proves the change does not blanket-catch OSError, and the ping path has its own EBADF case. All tests are hermetic (mock socket, no sleeps).
  • _is_transient_ws_error is a module-level helper rather than a method, which fits the existing module-level TRANSIENT_WS_ERRORS constant. It is not exported, so no public API change.
  • Considered but not flagged: _is_transient_ws_error is not applied to the except BrokenPipeError in _unsubscribe. That path already logs at WARNING and is outside this PR's scope; BrokenPipeError is already in TRANSIENT_WS_ERRORS anyway.
  • Considered but not flagged: OSError with errno=None (e.g. OSError("msg")) falls through to ERROR. That is the intended conservative behavior per the PR description.
  • The CHANGELOG entry and version bump to 3.10.1 follow the repo's release convention.

🤖 PR Reviewer · DeepSeek V4.1 Flash (Fireworks)

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

No code changes from this review. The findings section has nothing to act on, and there are no inline threads.

Merge stays blocked on the main ruleset, not on this diff. It requires one approving review, including a code owner, plus last-push approval. This review is a comment, not an approval, and I cannot approve my own PR.

CI that this change can affect is green: tests and check_outdated passed. CodeQL was still running when I checked. security/snyk (valiot) is in error with:

You have used your limit of private tests

That is an org quota, not a finding in this diff, and it is not a required check on the main ruleset. Nothing in the PR can clear it.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
What changed in this PR

Reduces error-level log noise during WSS subscription teardown by treating specific dead-socket OSError errnos as transient reconnect conditions (FEMSA-80).

Changes:

  • Add _is_transient_ws_error() to classify EBADF / ENOTCONN / bare ECONNRESET as transient WSS errors.
  • Downgrade logging from ERROR to WARNING for these transient teardown cases in _sub_routing_loop and _ping_pong.
  • Add regression tests and bump version + changelog for 3.10.1.
File Description
pygqlc/​GraphQLClient.py Adds transient OSError classification helper and uses it to adjust WSS teardown logging behavior.
tests/​pygqlc/​gql_client/​test_subscribe.py Adds tests asserting WARNING vs ERROR logging for dead-socket and non-dead-socket OSError cases.
pygqlc/​__version__.py Bumps library version to 3.10.1.
CHANGELOG.md Documents the behavior change and release notes for 3.10.1.

Review effort: Lite

Comment on lines 1 to 5
from . import subscriptions as subs
from . import mutations as muts
import errno
import types
import time
they must be classified as a transient teardown — WARNING and reconnect,
not ERROR with a traceback. ECONNRESET is already ConnectionResetError."""
gql = routing_client
gql._conn.recv.side_effect = OSError(errno_code, "Bad file descriptor")
@barucAlmaguer
barucAlmaguer merged commit 2c4a7e7 into main Nov 2, 2026
6 of 8 checks passed
@barucAlmaguer
barucAlmaguer deleted the palantir/FEMSA-80-transient-wss-oserror branch November 2, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants