Repository navigation
fix(auth): OAuth request lock, and keep sessions through transient restore failures (TECH-807) - #260
Conversation
NodeOAuthClient ran without a requestLock, so two Vercel instances could
refresh the same session at once. Refresh tokens rotate: the second
refresh replays a used token and the authorization server revokes the
session, signing the user out. The client warned about this on every
cold start ("No lock mechanism provided. Credentials might get revoked.").
The lock queues calls in-process first, then holds oauth:lock:<name> in
Redis (SET NX PX 45000) under a random owner token, released by an
owner-checked EVAL so a lock that lapsed and was taken over is left alone.
The TTL outlasts the client's 30s read timeout, because a configured lock
turns off the client's own concurrent-refresh recovery. A lock that stays
held past the wait throws RequestLockTimeoutError, whose message the
session route treats as transient rather than as a revoked session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Caution Review failedThe pull request is closed. Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The verify script only matched a service id of exactly "#atproto_pds". DID
documents may also use the full reference ("did:plc:...#atproto_pds"), and
those records failed with "no #atproto_pds service" before verification
started. Accept both forms; the app's resolvePdsUrl already does.
Found by Greptile on #254.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he request lock (TECH-807) Review follow-up to 6e443d1. Restore failures: - The xrpc proxy (reads and writes) and onboarding/clone-blob deleted the app session on any client.restore() error. A short outage fetching the authorization server's metadata, an Upstash error or a request-lock timeout therefore signed the user out; the 28 Oct staging sign-out followed exactly that metadata error on /api/xrpc getSession. - New isSessionGone(err, did) decides for every caller, including /api/auth/session. Revoked/invalid/unrefreshable tokens, 401/403 and the existing message check still delete the session and answer 401. Everything else keeps it and answers 503, which authFetch does not treat as signed out. - The client reports a failed session-store read as "The session was deleted by another process" (CachedGetter swallows store errors), the same as a really missing session, so that error is checked against Redis before deleting. Request lock: - The refresh request has no timeout and can outlast the TTL, letting a second instance replay the refresh token. The holder now renews the TTL every 5s (owner-checked PEXPIRE); the TTL drops to 15s and only bounds a crashed holder. Waiters still give up after 45s. - Poll from 5ms with backoff to 150ms instead of a flat 50-150ms; a typical holder keeps the lock for a couple of round trips. - The Upstash client retries a request whose response was lost; if the first SET landed, the retry saw our own key and waited on it. A null SET now checks whether the key holds our token. - Warnings go through logSafe. Tests: the fake Redis applies the scripts' owner check only when the script contains it; new cases for held-across-await, fresh tokens, lost SET responses, renewal and the wait boundary; unique lock names per test; wiring test moved to its own file; classifier tests; xrpc GET/POST and clone-blob tests for transient vs. revoked restores. AGENTS.md: document both, and drop the in-memory Redis fallback it described but stores.ts never had. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| const renew = setInterval(() => { | ||
| redis.eval(RENEW_SCRIPT, [key], [token, String(LOCK_TTL_MS)]).catch((err: unknown) => { | ||
| logSafe("[oauth-lock] renew failed", err) | ||
| }) | ||
| }, RENEW_EVERY_MS) |
There was a problem hiding this comment.
If Redis renewal fails for more than 15 seconds while a refresh is still running, the key expires and another instance can refresh the same token. The first holder keeps running: renewal errors only log, and a renewal result of 0 is ignored. The overlapping refreshes can replay a rotating token and revoke the session.
Bound the refresh work and handle lost ownership explicitly rather than continuing under an expired lock.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/auth/request-lock.ts
Line: 84-88
Comment:
**Refreshes outlive the lock**
If Redis renewal fails for more than 15 seconds while a refresh is still running, the key expires and another instance can refresh the same token. The first holder keeps running: renewal errors only log, and a renewal result of `0` is ignored. The overlapping refreshes can replay a rotating token and revoke the session.
Bound the refresh work and handle lost ownership explicitly rather than continuing under an expired lock.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| sessionStore: new RedisSessionStore(), | ||
| // Only one instance may refresh a DID's rotating refresh token at a | ||
| // time; a parallel refresh replays it and revokes the session. | ||
| requestLock: redisRequestLock, |
There was a problem hiding this comment.
Group timeouts delete valid sessions
Adding redisRequestLock introduces RequestLockTimeoutError into group requests, but getAuthenticatedAgent still calls deleteSession() after every restore error. If another instance holds the lock for 45 seconds, a request such as /api/groups/memberships deletes a valid app session and returns 401 instead of keeping it and returning 503.
Apply the shared error check to the group helper and preserve the temporary-failure response through its callers.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/auth/oauth-client.ts
Line: 122
Comment:
**Group timeouts delete valid sessions**
Adding `redisRequestLock` introduces `RequestLockTimeoutError` into group requests, but `getAuthenticatedAgent` still calls `deleteSession()` after every restore error. If another instance holds the lock for 45 seconds, a request such as `/api/groups/memberships` deletes a valid app session and returns 401 instead of keeping it and returning 503.
Apply the shared error check to the group helper and preserve the temporary-failure response through its callers.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Hotfix to
mainfor TECH-807, already onstaging(6e443d1, 29c5cf4). After merge, main and staging hold identical files.Problem
NodeOAuthClientran without arequestLock. Two Vercel instances could refresh the same session at once. Refresh tokens rotate, so the second refresh replays a used token, the authorization server revokes the session, and the user is signed out. The client logged "No lock mechanism provided. Credentials might get revoked." on every new instance.onboarding/clone-blobdeleted the app session on anyclient.restore()error. A short outage fetching the authorization server's metadata, an Upstash error or a lock timeout signed the user out. The 28 Oct staging sign-out followed exactly that metadata error on/api/xrpc/...getSession.Fix
Request lock (
src/lib/auth/request-lock.ts, passed asrequestLock)requestLocalLock). Across instances the lock isoauth:lock:<name>in the existing Upstash Redis:SET NX PX 15000with a random owner token, released by an owner-checkedEVAL.PEXPIRE). The token refresh has no timeout of its own, and with a lock configured the client no longer recovers from a concurrent refresh, so the lock must stay held for as long as the holder works. The 15s TTL only bounds how long a crashed holder blocks.RequestLockTimeoutError.SETreturns null but the key already holds our token (the Upstash client retried after a lost response), it counts as acquired.Restore failures (
src/lib/auth/restore-error.ts)isSessionGone(err, did)is used by/api/auth/session, both xrpc paths and clone-blob. Revoked/invalid/unrefreshable tokens, 401/403 and the existing message check delete the session and answer 401. Everything else keeps it and answers 503 (sessionUnavailable()), whichauthFetchdoes not treat as signed out.Also included:
scripts/verify-created-via.mjsaccepts a PDS service id given as the full DID URL (did:plc:...#atproto_pds). Found by Greptile on #254 after it merged; staging188008f7.Testing
fn, fresh tokens, waiting for another instance, not deleting a taken-over lock, release on throw, in-process queueing, lostSETresponses, renewal and its stop, and the 45s wait boundary. Each of these fails a test when broken on purpose: release without owner check, droppingawait fn(), a shorter wait, no renewal, a constant token, no self-acquire check, an uncleared renew timer.tsc,typecheck:test,lint,lint:css, 1423 unit tests,build.Not yet verified
The lock against real Upstash under a signed-in session (local
.env.localhas placeholder Upstash credentials). After a signed-in browse, check the logs forRequestLockTimeoutError,[oauth-lock] renew failedor[oauth-lock] release failed.Reviewed by three subagents (atproto client internals, production failure modes, tests); this revision addresses their findings.
🤖 Generated with Claude Code
Fix lost-lock behavior and group-route session deletion before merging.
Fix with agent prompt
Summary
Adds a Redis-backed OAuth lock and keeps app sessions through temporary restore failures in the XRPC and clone-blob routes. The verifier also accepts full-DID PDS service IDs.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Restore OAuth session] --> B[Local queue] B --> C[Acquire Redis lock] C --> D[Read session and refresh if needed] D --> E[Renew lock while working] D --> F[Release owned lock] C -->|45-second wait expires| G[RequestLockTimeoutError] G --> H[XRPC and clone-blob keep session] G --> I[Group helper deletes session] E -->|Renewal fails and key expires| J[Another instance can acquire] J --> K[Refresh work can overlap]Reviews (1) · Last reviewed commit: "fix(auth): keep sessions through transie..." · Reviewed by Greptile