Skip to content

fix: quieter, race-safe WSS ping failures (OPS-3616) (3.9.1) - #68

Merged
acrogenesis merged 5 commits into
mainfrom
palantir/OPS-3616-fix-ping-pong-connectionreset-log
Oct 4, 2026
Merged

acrogenesis merged 5 commits into
mainfrom
palantir/OPS-3616-fix-ping-pong-connectionreset-log

Conversation

@palantir-valiot

@palantir-valiot palantir-valiot Bot commented Jun 3, 2026 •

Copy link
Copy Markdown
Contributor

The ping thread treated every failed send as a broken pipe: it logged at ERROR with a full traceback on routine peer resets, and it could halt a healthy connection that the router thread had just reopened.

  • Transient send errors (TRANSIENT_WS_ERRORS) now log at WARNING, matching how recv errors have been logged since 3.7.1. The warning names the ping thread and includes the exception, so it doesn't look like a duplicate of the recv-side line. Unknown errors still log at ERROR.
  • The ping loop pins the socket it sends on. If the send fails after the router has already replaced that socket, the failure is ignored instead of setting wss_conn_halted, which used to force one extra reconnect. While _conn is None mid-reconnect, the ping loop skips the send instead of logging an AttributeError at ERROR.
  • Reconnect behavior is otherwise unchanged. The 3.9.1 bump is included.

Regression tests cover both races and fail against the old loop.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SxDqwJXiewNEGu7kjnJQ5m

💬 Continue the agent chat (live transcript, tools, follow-ups):
https://palantir.valiot.io/agent-tasks?task=6a64d681-e037-42e2-ab63-4fbe6c887bf5

@linear-code

linear-code Bot commented Jun 3, 2026

Copy link
Copy Markdown

OPS-3616

@palantir-valiot
palantir-valiot Bot force-pushed the palantir/OPS-3616-fix-ping-pong-connectionreset-log branch from 50a752e to 8e13ee8 Compare June 12, 2026 04:31

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

The PR applies the same transient-vs-fatal error classification already present in _sub_routing_loop (OPS-3485) to the _ping_pong send path. ConnectionResetError/BrokenPipeError/etc. now emit WARNING instead of ERROR+traceback; reconnection via wss_conn_halted is unchanged. Implementation mirrors the recv pattern exactly, tests are deterministic and fast, and release metadata is updated. No blocking bugs found.

Findings

No actionable findings.

Notes

  • The change is isolated to a single if isinstance(e, TRANSIENT_WS_ERRORS) branch inside the existing bare except Exception in _ping_pong, preserving all prior behavior for non-transient errors.
  • Tests reuse the exact _capture_logs + MagicMock + side-effect termination pattern from the OPS-3485 recv tests; both new tests pass the TDD red-then-green criterion described in the PR body.
  • TRANSIENT_WS_ERRORS is already a module-level tuple (line 35) and is referenced identically on both recv (line 546) and send (line 784) paths.

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

All checks are green (tests, analyze, CodeQL, check_outdated, Snyk). The Palantir PR Reviewer assessment is positive with "No actionable findings."

No code changes required. The implementation mirrors the OPS-3485 recv classification for transient send errors in _ping_pong, the new tests are deterministic/fast and pass the TDD red/green bar, and release metadata (version + CHANGELOG) is updated per AGENTS.md.

Ready for maintainer merge.

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

Rebase recovery during git push-safe hit a conflict replaying recent main commits (b8a1958 etc.) onto the old remote PR tip (d4ea664). Working tree is clean; our change is still a real diff vs current origin/main:

pygqlc/GraphQLClient.py
+ if isinstance(e, TRANSIENT_WS_ERRORS):
+     log(WARNING, ...)
+ else:
+     log(ERROR, ...)

The bug is not yet fixed on main (main's _ping_pong still unconditionally logs ERROR on send exceptions). This is a mechanical CHANGELOG rebase conflict on the PR branch history, not a semantic conflict in the fix.

Per instructions, escalating for human resolution of the push/rebase. The candidate commit on the local branch is 6e821d9 (version 3.8.4 + tests + the send-side transient handling).

@palantir-valiot
palantir-valiot Bot force-pushed the palantir/OPS-3616-fix-ping-pong-connectionreset-log branch from d4ea664 to 6e821d9 Compare June 24, 2026 18:55
@palantir-valiot
palantir-valiot Bot requested a review from acrogenesis June 24, 2026 18:55

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

This PR fixes noisy ERROR+traceback logs in _ping_pong when transient connection errors (ConnectionResetError, BrokenPipeError, etc.) occur during the keep-alive ping send. The fix mirrors the exact classification pattern already used in _sub_routing_loop recv (OPS-3485). No blocking bugs found.

Findings

No actionable findings.

Notes

  • TDD followed per AGENTS.md: tests written first (red because send path always emitted ERROR), then made green by the isinstance(e, TRANSIENT_WS_ERRORS) guard.
  • Tests are hermetic (<100 ms, no sockets, patch("time.sleep") + side-effect termination) and modeled exactly on the existing recv regression tests.
  • TRANSIENT_WS_ERRORS constant (defined at module level, lines 35-40) is reused; the log message for transient cases matches the recv path exactly.
  • Version bump, CHANGELOG entry, and CI checks (format/bandit/pytest/lock) all completed as required.

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

Thanks for the review. All checks green (our ping-pong regression tests + ruff format + bandit). No further changes needed from our side. The current BLOCKED state is CodeQL in-progress + required review; ready when those clear.

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

❌ Agent run failed

The implementer agent encountered a failure and has been parked
in the failed state. The PR remains open for human follow-up.

Reason: "opencode APIError: Forbidden: You have run out of credits or need a Grok subscription. Add credits at https://grok.com/?_s=usage or upgrade at https://grok.com/supergrok."

If this is a transient provider error (e.g. xAI capacity), the
automatic retry sweeper will revive the run within ~15 min.
Otherwise, reply on this PR or the Linear thread with context
and the run will resume.

@acrogenesis
acrogenesis force-pushed the palantir/OPS-3616-fix-ping-pong-connectionreset-log branch from 6e821d9 to dfbfa7e Compare October 4, 2026 17:53
Copilot AI balanced review requested due to automatic review settings October 4, 2026 17:53
@acrogenesis acrogenesis changed the title Fix ping-pong send ConnectionResetError noise at ERROR level (OPS-3616) fix: log transient ping-send errors at WARNING (OPS-3616) (3.9.1) Oct 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 Medium severity · 1 Low severity

Open (3)
What changed in this PR

Adjusts _ping_pong keep-alive logging to treat transient websocket send failures as WARNING (instead of ERROR) to reduce expected-noise during WSS churn while preserving reconnect behavior.

Changes:

  • Classifies TRANSIENT_WS_ERRORS raised by self._conn.send(PING_JSON) as WARNING in _ping_pong.
  • Adds unit tests covering transient vs unexpected send failures and expected log levels.
  • Bumps package version and documents the change in CHANGELOG.md.
File Description
pygqlc/​GraphQLClient.py Downgrades log level for transient ping-send failures while still halting the WSS connection.
tests/​pygqlc/​gql_client/​test_subscribe.py Adds regression tests ensuring correct log level + halt behavior for ping-send exceptions.
pygqlc/​__version__.py Bumps version to 3.9.1.
CHANGELOG.md Adds release entry documenting the change.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pygqlc/__version__.py Resolved
Comment thread pygqlc/GraphQLClient.py Resolved
Copilot AI balanced review requested due to automatic review settings October 4, 2026 18:12
…ced socket

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxDqwJXiewNEGu7kjnJQ5m
@acrogenesis acrogenesis changed the title fix: log transient ping-send errors at WARNING (OPS-3616) (3.9.1) fix: quieter, race-safe WSS ping failures (OPS-3616) (3.9.1) Oct 4, 2026

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

_ping_pong now logs TRANSIENT_WS_ERRORS from send at WARNING and still sets wss_conn_halted. That matches the recv path and drops the ERROR traceback for a peer reset. The classification is correct for websocket-client: ConnectionResetError, BrokenPipeError, and ConnectionAbortedError propagate from sock.send, and a missing socket raises WebSocketConnectionClosedException.

The new tests can exit before send runs, so they can fail on a fast clock tick rather than on the log level.

Findings

# Severity File Issue
1 Low tests/pygqlc/gql_client/test_subscribe.py Ping tests require wall-clock time to advance before the loop is closed, so they can miss send and fail.

Notes

  • WARNING is the right level. pygqlc.logging prints a traceback only for ERROR, in both the fallback logger and valiotlogging.
  • Reconnect behavior is unchanged: a send failure still sets wss_conn_halted.
  • Considered flagging a missing OSError / TimeoutError on send. websocket-client re-raises the connection subclasses above, and a send timeout was already ERROR before this change. Out of scope for OPS-3616.
  • The PR body says the bump is 3.8.0 → 3.8.1 and that uv.lock gained idna/snowballstemmer. The diff is 3.9.0 → 3.9.1 and does not touch uv.lock. Version and changelog agree with each other.

🤖 PR Reviewer · Grok 4.7 (xAI)

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

_ping_pong now classifies TRANSIENT_WS_ERRORS at WARNING and skips a send when _conn is None. That matches the recv-side handling from 3.7.1, and the new tests cover the intended cases. The identity check meant to ignore a replaced socket does not hold: the router can swap self._conn back to the same object before the ping thread evaluates conn is self._conn, so a dead ping still sets wss_conn_halted on a healthy reconnect. The PR also claims a committed uv lock --upgrade that is not in the diff.

Findings

# Severity File Issue
1 Medium pygqlc/GraphQLClient.py conn is self._conn is true again after the router reconnects to the same socket object, so the guard fails to stop the extra halt
2 Low CHANGELOG.md The PR body says uv lock --upgrade was committed, but uv.lock is not in the diff

Notes

  • The WARNING path drops the traceback. pygqlc/logging.py only prints traceback.format_exc() for LogLevel.ERROR, and the warning message includes e!r. That is the noise reduction OPS-3616 asks for.
  • Considered flagging the missing websocket import in the test file. The file already imports it at line 11; the diff only shows the new use.
  • test_ping_pong_send_failure_on_replaced_socket_does_not_halt only models a swap that sticks. It does not model the router finishing _new_conn and restoring the same socket before the except block runs.
  • The conn is None skip is sound. _close_conn assigns None before create_connection, and a None send would previously raise AttributeError and log at ERROR.

🤖 PR Reviewer · Grok 4.7 (xAI)

Comment thread pygqlc/GraphQLClient.py Resolved
Comment thread CHANGELOG.md Resolved
@acrogenesis

Copy link
Copy Markdown
Member

@Eduardotrom your review's items are addressed: the branch is rebased onto 3.9.0 and bumped to 3.9.1 (no version collision), the stale PR body is rewritten, and the CHANGELOG entry is re-stacked without the stray blank line. Since then the PR also includes the ping/reconnect race fix (7bddea3) and clock-independent tests (05d4139). Could you take another look?

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

All six review threads are resolved. CI is green on 05d4139 (tests, analyze, CodeQL, check_outdated, Snyk). No further code change.

The identity check holds. _new_conn always assigns a new WebSocket from create_connection, so a send that raises after the router swaps sockets sees conn is self._conn as false and does not set wss_conn_halted. The test test_ping_pong_send_failure_on_replaced_socket_does_not_halt covers that order. The leftover window is the check and the flag assignment, which are adjacent; a reconnect landing between them causes one extra reconnect. A generation counter has the same gap. Closing it needs a lock shared with the router, which is not worth it for a harmless reconnect.

Merge stays blocked on the ruleset: one approving review and last-push approval. Eduardotrom's June CHANGES_REQUESTED is still the open review. The version collision, stale body, and changelog nit from that review are already fixed on this branch (3.9.1, rebased on 3.9.0).

@palantir-valiot
palantir-valiot Bot requested a review from doctorcorral October 4, 2026 18:24
@acrogenesis
acrogenesis merged commit fe1dd2b into main Oct 4, 2026
7 checks passed
@acrogenesis
acrogenesis deleted the palantir/OPS-3616-fix-ping-pong-connectionreset-log branch October 4, 2026 18:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 1 High severity · 1 Medium severity

Open (2)
Resolved since last review (3)


@pytest.fixture
def routing_client():
"""A fresh GraphQLClient (bypassing the process-wide singleton cache) wired to a mock
Comment thread CHANGELOG.md

## [3.9.1] - 2026-10-04

- [Fixed] `_ping_pong` now logs a transient send failure (`TRANSIENT_WS_ERRORS`: connection reset/aborted, broken pipe, websocket closed) at WARNING instead of ERROR, matching how `_sub_routing_loop` has classified the same errors on `recv` since 3.7.1. Reconnect via `wss_conn_halted` is unchanged; only the log level differs. The warning names the ping thread and includes the exception, so it is distinguishable from the recv-side warning. (OPS-3616)
Copilot AI balanced review requested due to automatic review settings October 4, 2026 18:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if alive:
# Stop it so a stuck loop can't leak into later tests.
gql.closing = True
thread.join(timeout)
Comment thread CHANGELOG.md
Comment on lines +3 to +5
## [3.9.1] - 2026-10-04

- [Fixed] `_ping_pong` now logs a transient send failure (`TRANSIENT_WS_ERRORS`: connection reset/aborted, broken pipe, websocket closed) at WARNING instead of ERROR, matching how `_sub_routing_loop` has classified the same errors on `recv` since 3.7.1. Reconnect via `wss_conn_halted` is unchanged; only the log level differs. The warning names the ping thread and includes the exception, so it is distinguishable from the recv-side warning. (OPS-3616)
Comment thread pygqlc/GraphQLClient.py
Comment on lines 842 to +847
self.pingTimer = current_time
# The router thread may swap self._conn mid-send while reconnecting;
# pin the socket so a failure on a replaced one is recognizable.
conn = self._conn
if conn is None:
continue
Comment thread pygqlc/GraphQLClient.py
Comment on lines +856 to +865
if isinstance(e, TRANSIENT_WS_ERRORS):
log(
LogLevel.WARNING,
f"WSS ping send failed, connection closed by peer: {e!r}",
)
else:
log(
LogLevel.ERROR,
"error trying to send ping, WSS Pipe is broken",
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants