Skip to content

fix: resend a request only when it never reached the server (3.9.2) - #86

Merged
acrogenesis merged 3 commits into
mainfrom
acrogenesis/safe-repost
Oct 4, 2026
Merged

acrogenesis merged 3 commits into
mainfrom
acrogenesis/safe-repost

Conversation

@acrogenesis

Copy link
Copy Markdown
Member

pygqlc re-sent a request after a transport failure even when the server had already received it. Sync execute re-POSTed once after any exception; async_execute did so after any NetworkError or RemoteProtocolError. When only the answer was lost (a read timeout, or the connection dropped after the server committed), a mutation without a unique key was silently stored twice. This happened in practice in the Prolamsa requirements upload (valiot/prolamsa-worker#612, INTM-231): a lost answer on a createBulkOrderSupplyAllocations chunk stored 500 allocations twice. The job reported success, and the sequencer then read doubled precedence.

  • New rule: a request is re-sent only if it never fully left the client. An httpcore trace callback records send_request_body.complete, and pygqlc retries only when that event never fired, or when the client or event loop was closed (as before).
  • Still recovers transparently: refused or timed-out connects, pool timeouts, and pooled connections the server killed while idle. That includes HTTP/2: httpcore doesn't health-check h2 connections before reuse, so a write to a dead socket raises WriteError.
  • Now raised to the caller: any failure after the request was written (ReadError, ReadTimeout, RemoteProtocolError, a lost answer).
  • Same rule for queries and mutations. execute can't tell them apart without parsing the document. Callers that relied on silent retries of reads after the server received them now see the error (CHANGELOG [Changed]).
  • Sync client leak fixed: sync execute no longer replaces and leaks its thread-local client on every failure.
  • Version 3.9.2 (patch).

Tests. tests/pygqlc/gql_client/test_resend_policy.py has 31 tests. They run against real local servers, an HTTP/1.1 keep-alive server and an HTTP/2 prior-knowledge server, which count POSTs and connections. No mocks.

  • Refused first, then up: succeeds with exactly 1 POST.
  • Commit then drop, reset, or stall past post_timeout: the error surfaces with exactly 1 POST.
  • Keep-alive reuse and idle close behave as expected.
  • HTTP/2 connections killed while idle (FIN, RST, GOAWAY+FIN) recover.
  • Each case runs for both sync and async.

Red/green: the duplicate tests fail on 3.9.1, the HTTP/2 stale-connection tests fail on the first draft (53d54b5), and all pass on this head. The tests were stable over 20 sequential and 30 parallel runs. The full suite gives 85 passed; the 25 live tests that need API/WSS/TOKEN were not run.

Known residual risk (theoretical): with async HTTP/2, another stream on the same connection could flush this request's END_STREAM, and a later failure in that shared write would resend a request the server may have received. A unique key on the server side remains the complete fix for non-idempotent creates.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VBSS7jE8JNJSWSz1ti8xVi

acrogenesis and others added 2 commits October 4, 2026 13:12
A mutation without a unique key (e.g. a bulk create) was stored twice when the
server committed it but the answer was lost: sync execute re-POSTed after any
exception, async_execute after any ReadError/RemoteProtocolError.

- Resend only after ConnectError, ConnectTimeout, PoolTimeout or a closed
  client/event loop, for sync and async alike; every other transport failure
  is raised to the caller (query/mutate report it in errors).
- One rule for queries and mutations: execute cannot tell them apart, and the
  stale keep-alive case 3.8.4 targeted is already handled by httpcore.
- Tests drive a real local HTTP server that refuses, drops, resets or stalls,
  and count the POSTs it received.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBSS7jE8JNJSWSz1ti8xVi
…ully written

The first 3.9.2 change turned a stale pooled HTTP/2 connection into an error:
httpcore checks an HTTP/1.1 connection before reusing it but not an HTTP/2 one,
so a connection the server closed without a readable GOAWAY (crash, killed pod,
LB reset) within the 5 s keep-alive window failed with WriteError, where 3.9.1
had resent it.

- Decide with an httpcore trace callback instead of a fixed error list: resend
  after any TransportError raised before send_request_body.complete (end of
  body / END_STREAM). A server cannot run a request it never fully received;
  any failure after that point is still raised, so no mutation runs twice.
- The idle-close tests now really use HTTP/1.1 keep-alive and count
  connections; a real h2 server covers the stale-connection resend and the
  commit-then-drop case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBSS7jE8JNJSWSz1ti8xVi
Copilot AI balanced review requested due to automatic review settings October 4, 2026 19:46
@acrogenesis acrogenesis self-assigned this Oct 4, 2026
palantir-valiot[bot]
palantir-valiot Bot previously approved these changes Oct 4, 2026

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

execute and async_execute now resend only when the request never left the client, which stops the silent double-write of a mutation whose answer was lost. The implementation matches httpcore's contract: send_request_body.complete is emitted from Trace.__exit__ only after _write_outgoing_data / the HTTP/1.1 body write returns, so a WriteError still retries and a failure after the write does not. No blocking bugs found.

Findings

No actionable findings.

Notes

  • The live tests cover the failure that caused INTM-231: commit-then-drop, RST, and a stall past post_timeout each raise with exactly one POST, for both sync and async, plus the HTTP/2 idle-close cases that the first draft missed.
  • Considered flagging the retry POST omitting extensions={"trace": trace}. It is harmless: the second attempt is not retried, and a second Trace on the same callback would only set sent again.
  • Considered flagging RuntimeError matching on str(error) ("Event loop is closed", "client has been closed"). That predicate is unchanged from 3.8.x and still covered by test_should_retry_on_fresh_connection and the closed-loop rebuild test.
  • The shared-connection residual risk in the PR body is real but narrow: on async HTTP/2, _write_outgoing_data drains every buffered frame under _write_lock, so another stream can flush this request's END_STREAM and then fail the shared write before this stream's send_request_body.complete. A unique server-side key is still the complete fix, as noted.
  • AGENTS.md says tests should use no real sockets and no sleeps. These tests need both to prove the wire behavior, and the PR says they were stable over 20 sequential and 30 parallel runs. Not flagged.

🤖 PR Reviewer · Grok 4.7 (xAI)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 Medium severity

Open (2)
What changed in this PR

Updates pygqlc’s resend behavior to avoid duplicating mutations when the server may already have received the request, using httpcore tracing to retry only when the request body was not fully written.

Changes:

  • Add httpcore trace-based “request fully sent” detection and gate retries on it for both sync and async execution.
  • Replace/expand prior mocked retry tests with real HTTP/1.1 and HTTP/2 local-server integration tests validating POST/connection counts.
  • Bump version to 3.9.2 and document the behavioral change in the changelog.
File Description
pygqlc/​GraphQLClient.py Implements trace-based resend policy and updates retry predicate/signature.
tests/​pygqlc/​gql_client/​test_resend_policy.py Adds end-to-end tests using real local HTTP/1.1 and HTTP/2 servers to verify resend policy.
tests/​pygqlc/​gql_client/​test_transient_transport_retry.py Removes older mock-based tests superseded by the new resend policy tests.
pygqlc/​__version__.py Bumps library version to 3.9.2.
CHANGELOG.md Documents new resend rule and the user-visible behavior change.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…eping

The idle-close tests slept a fixed 0.3 s and assumed the server had closed
its idle connection by then; a slow CI runner could overrun that and fail the
test. Both test servers now set an event once they close the connection, and
the tests wait on it (10 s cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBSS7jE8JNJSWSz1ti8xVi
Copilot AI balanced review requested due to automatic review settings October 4, 2026 21:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (2)

Comment thread pygqlc/GraphQLClient.py Resolved
Copilot AI balanced review requested due to automatic review settings October 4, 2026 21:46
@acrogenesis
acrogenesis merged commit faf74e0 into main Oct 4, 2026
6 of 8 checks passed
@acrogenesis
acrogenesis deleted the acrogenesis/safe-repost branch October 4, 2026 21:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 Low severity

Open (2)
Resolved since last review (1)

Comment thread CHANGELOG.md
Comment on lines +3 to 8
## [3.9.2] - 2026-10-04

- [Fixed] `execute` and `async_execute` send a request again only when it provably never reached the server: a transport failure (`httpx.TransportError`) raised before the request was written in full, or a closed client / closed or foreign event loop (as before). An httpcore `trace` callback records `send_request_body.complete` (the end of the body, or HTTP/2 END_STREAM); a server cannot run a request whose last bytes never left the client. That covers a refused or timed-out connect, a pool timeout, and a pooled HTTP/2 connection the server had already closed without a readable GOAWAY (a crash, a killed pod, a load-balancer reset); httpcore checks an HTTP/1.1 pooled connection before reusing it, but not an HTTP/2 one, so the request is written to the dead socket and fails with `WriteError`. Any failure after the request was written (`ReadError`, `ReadTimeout`, `RemoteProtocolError`, a lost answer) is now raised to the caller, so `query`/`mutate` report it in `errors`. Previously sync `execute` re-POSTed once after ANY exception and `async_execute` after any `httpx.NetworkError` or `RemoteProtocolError`; when the server had already committed the first request and only its answer was lost (connection dropped or reset, or a sync read timeout), a mutation without a unique key — e.g. a bulk create — was silently stored twice.
- [Changed] One rule for queries and mutations: `execute` cannot tell a read from a write without parsing the document. A query that fails after reaching the server (its answer lost, a read timeout, or the server closing the connection after the request was written) is no longer resent; the caller has to retry it. A duplicated mutation is not cheap to undo. Sync `execute` also stops replacing (and leaking) its thread-local client on every failure: like `async_execute`, it resends on the same pool, and `_get_http_client` replaces a closed client. `TRANSIENT_TRANSPORT_ERRORS` is removed; `_should_retry_on_fresh_connection` takes a second `request_sent` argument.

## [3.9.1] - 2026-10-04
Comment thread CHANGELOG.md
- [Fixed] `execute` and `async_execute` send a request again only when it provably never reached the server: a transport failure (`httpx.TransportError`) raised before the request was written in full, or a closed client / closed or foreign event loop (as before). An httpcore `trace` callback records `send_request_body.complete` (the end of the body, or HTTP/2 END_STREAM); a server cannot run a request whose last bytes never left the client. That covers a refused or timed-out connect, a pool timeout, and a pooled HTTP/2 connection the server had already closed without a readable GOAWAY (a crash, a killed pod, a load-balancer reset); httpcore checks an HTTP/1.1 pooled connection before reusing it, but not an HTTP/2 one, so the request is written to the dead socket and fails with `WriteError`. Any failure after the request was written (`ReadError`, `ReadTimeout`, `RemoteProtocolError`, a lost answer) is now raised to the caller, so `query`/`mutate` report it in `errors`. Previously sync `execute` re-POSTed once after ANY exception and `async_execute` after any `httpx.NetworkError` or `RemoteProtocolError`; when the server had already committed the first request and only its answer was lost (connection dropped or reset, or a sync read timeout), a mutation without a unique key — e.g. a bulk create — was silently stored twice.
- [Changed] One rule for queries and mutations: `execute` cannot tell a read from a write without parsing the document. A query that fails after reaching the server (its answer lost, a read timeout, or the server closing the connection after the request was written) is no longer resent; the caller has to retry it. A duplicated mutation is not cheap to undo. Sync `execute` also stops replacing (and leaking) its thread-local client on every failure: like `async_execute`, it resends on the same pool, and `_get_http_client` replaces a closed client. `TRANSIENT_TRANSPORT_ERRORS` is removed; `_should_retry_on_fresh_connection` takes a second `request_sent` argument.

## [3.9.1] - 2026-10-04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants