fix(api-client): retry rate-limited requests after Retry-After - #388
Merged
Merged
Conversation
A 429 was returned as is, so one throttled request failed the whole upload on sharded CI runs: "APIError: HTTP 429 Too Many Requests: Too many requests, please try again later." apiFetch now waits for Retry-After and retries, up to 5 times, waiting at most 60s each time. The API counts requests in fixed 5-minute windows, so this sees a request through a whole window while bounding how long CI waits. A missing or invalid header falls back to exponential backoff from minTimeout. These retries are counted apart from the server error retries, keep the request id and bump x-argos-retry-attempt, and aborting the request stops the wait. Once they run out, the last 429 response is returned unchanged, so callers report the same APIError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What
apiFetchretries a429 Too Many Requestsafter waiting for itsRetry-After, up to 5 times, waiting at most 60s each time whatever the header asks.Retry-Afteris missing or invalid (anything but a whole number of seconds, which is what the API sends), it backs off exponentially fromminTimeout: 1s, 2s, 4s, …x-argos-request-idand sends the nextx-argos-retry-attempt, counted across both kinds of retry.APIError. The 5xx path is unchanged.Why
apiFetchreturned any 429 as is, so one throttled request failed the whole upload:upload()threw and the Playwright/Vitest reporters printed❌ Error while creating the Argos build — APIError: HTTP 429 Too Many Requests: Too many requests, please try again later.A customer hit this on sharded CI runs.The API's rate limiter (express-rate-limit with draft-8 headers and a Redis store) sends
Retry-After: <seconds>on every 429, and counts requests in fixed 5-minute windows that start at the first request. A burst from an otherwise idle IP therefore usually gets aRetry-Afterclose to the full 300s. Five waits of at most a minute is the smallest bound that still sees a request through a whole window. In exchange, rate limiting can hold a request up for 5 minutes at most, so CI can't hang on it.This goes with the backend moving the SDK's CI upload endpoints to their own, larger rate-limit budget (argos-ci/argos,
apps/backend/src/web/api/v2.ts): a burst over a limit should delay an upload rather than fail it.Type of changes
Bug fix (
buglabel).Checklist
Optional checks:
Further comments
Testing
packages/api-client/src/fetch.test.ts, all on fake timers: the retry onceRetry-Afterelapses (POST body replayed), the 60s cap, the fallback for a missing, empty, non-numeric or negative header, the last 429 returned after 5 retries (6 requests, 5 minutes in all), counting apart from server-error retries (headers checked), the 429 count surviving a server-error retry, and an abort during the wait. They all fail on the previous implementation, and breaking each part (cap, retry count, abort, fallback, counters) fails the matching test.express-rate-limitsetup: a request over the limit gotRetry-After: 3, waited 3s and succeeded; a request that was always rejected gave up after 6 attempts with the sameHTTP 429 Too Many Requests: Too many requests, please try again later.message.pnpm run test(all packages),pnpm run lint,pnpm run check-typesandpnpm run check-formatpass. The api-client tests also pass on Node 22, 24 and 26.For reviewers
createClient, including the CLI's interactive commands: a 429 there now waits up to 5 minutes, logged only underDEBUG=@argos-ci/api-client, instead of failing fast. A visible warning or a per-client setting could follow.Retry-AfterHTTP date is treated as invalid on purpose, since the API only sends seconds.🤖 Generated with Claude Code