Skip to content

fix(auth): OAuth request lock, and keep sessions through transient restore failures (TECH-807) - #260

Merged
holkexyz merged 3 commits into
mainfrom
holke/tech-807-oauth-request-lock
Oct 29, 2026
Merged

holkexyz merged 3 commits into
mainfrom
holke/tech-807-oauth-request-lock

Conversation

@svc-agent

@svc-agent svc-agent commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Hotfix to main for TECH-807, already on staging (6e443d1, 29c5cf4). After merge, main and staging hold identical files.

Problem

  1. NodeOAuthClient ran without a requestLock. Two Vercel instances could refresh the same session at once. Refresh tokens rotate, so the second refresh replays a used token, the authorization server revokes the session, and the user is signed out. The client logged "No lock mechanism provided. Credentials might get revoked." on every new instance.
  2. Found in review: the xrpc proxy (reads and writes) and onboarding/clone-blob deleted the app session on any client.restore() error. A short outage fetching the authorization server's metadata, an Upstash error or a lock timeout signed the user out. The 28 Oct staging sign-out followed exactly that metadata error on /api/xrpc/...getSession.

Fix

Request lock (src/lib/auth/request-lock.ts, passed as requestLock)

  • Calls on one instance queue in-process (requestLocalLock). Across instances the lock is oauth:lock:<name> in the existing Upstash Redis: SET NX PX 15000 with a random owner token, released by an owner-checked EVAL.
  • The holder renews the TTL every 5s (owner-checked PEXPIRE). The token refresh has no timeout of its own, and with a lock configured the client no longer recovers from a concurrent refresh, so the lock must stay held for as long as the holder works. The 15s TTL only bounds how long a crashed holder blocks.
  • Polling starts at 5ms and backs off to 150ms. Waiters give up after 45s with RequestLockTimeoutError.
  • If a SET returns null but the key already holds our token (the Upstash client retried after a lost response), it counts as acquired.

Restore failures (src/lib/auth/restore-error.ts)

  • isSessionGone(err, did) is used by /api/auth/session, both xrpc paths and clone-blob. Revoked/invalid/unrefreshable tokens, 401/403 and the existing message check delete the session and answer 401. Everything else keeps it and answers 503 (sessionUnavailable()), which authFetch does not treat as signed out.
  • The client reports a failed session-store read as "The session was deleted by another process", the same error as a really missing session. That one error is checked against Redis before deleting.

Also included: scripts/verify-created-via.mjs accepts a PDS service id given as the full DID URL (did:plc:...#atproto_pds). Found by Greptile on #254 after it merged; staging 188008f7.

Testing

  • Lock tests with a fake Redis that applies the scripts' owner check only when the script contains it. They cover holding across an async fn, fresh tokens, waiting for another instance, not deleting a taken-over lock, release on throw, in-process queueing, lost SET responses, renewal and its stop, and the 45s wait boundary. Each of these fails a test when broken on purpose: release without owner check, dropping await fn(), a shorter wait, no renewal, a constant token, no self-acquire check, an uncleared renew timer.
  • Classifier tests, plus xrpc GET/POST and clone-blob tests for transient vs. revoked restores. The transient cases fail on the old routes.
  • tsc, typecheck:test, lint, lint:css, 1423 unit tests, build.
  • Staging (6e443d1): the deployment with the lock builds the client and logs no "No lock mechanism provided".

Not yet verified

The lock against real Upstash under a signed-in session (local .env.local has placeholder Upstash credentials). After a signed-in browse, check the logs for RequestLockTimeoutError, [oauth-lock] renew failed or [oauth-lock] release failed.

Reviewed by three subagents (atproto client internals, production failure modes, tests); this revision addresses their findings.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 3/5

Fix lost-lock behavior and group-route session deletion before merging.

Fix All in Claude CodeFindings

  1. P1 Refreshes outlive the lock ▶
  2. P1 Group timeouts delete valid sessions ▶
Fix with agent prompt
### Issue 1
src/lib/auth/request-lock.ts:84-88
If Redis renewal fails for more than 15 seconds while a refresh is still running, the key expires and another instance can refresh the same token. The first holder keeps running: renewal errors only log, and a renewal result of `0` is ignored. The overlapping refreshes can replay a rotating token and revoke the session.

Bound the refresh work and handle lost ownership explicitly rather than continuing under an expired lock.

### Issue 2
src/lib/auth/oauth-client.ts:122
Adding `redisRequestLock` introduces `RequestLockTimeoutError` into group requests, but `getAuthenticatedAgent` still calls `deleteSession()` after every restore error. If another instance holds the lock for 45 seconds, a request such as `/api/groups/memberships` deletes a valid app session and returns 401 instead of keeping it and returning 503.

Apply the shared error check to the group helper and preserve the temporary-failure response through its callers.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a Redis-backed OAuth lock and keeps app sessions through temporary restore failures in the XRPC and clone-blob routes. The verifier also accepts full-DID PDS service IDs.

  • OAuth session restores wait their turn across server instances.
  • Temporary restore failures no longer sign users out.
  • The verification script finds a PDS by either service-ID form.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Restore OAuth session] --> B[Local queue]
  B --> C[Acquire Redis lock]
  C --> D[Read session and refresh if needed]
  D --> E[Renew lock while working]
  D --> F[Release owned lock]
  C -->|45-second wait expires| G[RequestLockTimeoutError]
  G --> H[XRPC and clone-blob keep session]
  G --> I[Group helper deletes session]
  E -->|Renewal fails and key expires| J[Another instance can acquire]
  J --> K[Refresh work can overlap]
Loading

Reviews (1) · Last reviewed commit: "fix(auth): keep sessions through transie..." · Reviewed by Greptile

NodeOAuthClient ran without a requestLock, so two Vercel instances could
refresh the same session at once. Refresh tokens rotate: the second
refresh replays a used token and the authorization server revokes the
session, signing the user out. The client warned about this on every
cold start ("No lock mechanism provided. Credentials might get revoked.").

The lock queues calls in-process first, then holds oauth:lock:<name> in
Redis (SET NX PX 45000) under a random owner token, released by an
owner-checked EVAL so a lock that lapsed and was taken over is left alone.
The TTL outlasts the client's 30s read timeout, because a configured lock
turns off the client's own concurrent-refresh recovery. A lock that stays
held past the wait throws RequestLockTimeoutError, whose message the
session route treats as transient rather than as a revoked session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
certified-app Ready Ready Preview Oct 29, 2026 1:17pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The verify script only matched a service id of exactly "#atproto_pds". DID
documents may also use the full reference ("did:plc:...#atproto_pds"), and
those records failed with "no #atproto_pds service" before verification
started. Accept both forms; the app's resolvePdsUrl already does.

Found by Greptile on #254.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he request lock (TECH-807)

Review follow-up to 6e443d1.

Restore failures:
- The xrpc proxy (reads and writes) and onboarding/clone-blob deleted the
  app session on any client.restore() error. A short outage fetching the
  authorization server's metadata, an Upstash error or a request-lock
  timeout therefore signed the user out; the 28 Oct staging sign-out
  followed exactly that metadata error on /api/xrpc getSession.
- New isSessionGone(err, did) decides for every caller, including
  /api/auth/session. Revoked/invalid/unrefreshable tokens, 401/403 and the
  existing message check still delete the session and answer 401.
  Everything else keeps it and answers 503, which authFetch does not treat
  as signed out.
- The client reports a failed session-store read as "The session was
  deleted by another process" (CachedGetter swallows store errors), the
  same as a really missing session, so that error is checked against Redis
  before deleting.

Request lock:
- The refresh request has no timeout and can outlast the TTL, letting a
  second instance replay the refresh token. The holder now renews the TTL
  every 5s (owner-checked PEXPIRE); the TTL drops to 15s and only bounds a
  crashed holder. Waiters still give up after 45s.
- Poll from 5ms with backoff to 150ms instead of a flat 50-150ms; a
  typical holder keeps the lock for a couple of round trips.
- The Upstash client retries a request whose response was lost; if the
  first SET landed, the retry saw our own key and waited on it. A null SET
  now checks whether the key holds our token.
- Warnings go through logSafe.

Tests: the fake Redis applies the scripts' owner check only when the
script contains it; new cases for held-across-await, fresh tokens, lost
SET responses, renewal and the wait boundary; unique lock names per test;
wiring test moved to its own file; classifier tests; xrpc GET/POST and
clone-blob tests for transient vs. revoked restores. AGENTS.md: document
both, and drop the in-memory Redis fallback it described but stores.ts
never had.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@svc-agent svc-agent changed the title fix(auth): give the OAuth client a Redis request lock (TECH-807) fix(auth): OAuth request lock, and keep sessions through transient restore failures (TECH-807) Oct 29, 2026
@holkexyz
holkexyz marked this pull request as ready for review October 29, 2026 13:30
@holkexyz
holkexyz merged commit 17b3a70 into main Oct 29, 2026
7 checks passed
@holkexyz
holkexyz deleted the holke/tech-807-oauth-request-lock branch October 29, 2026 13:30
svc-agent pushed a commit that referenced this pull request Oct 29, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +84 to +88
const renew = setInterval(() => {
redis.eval(RENEW_SCRIPT, [key], [token, String(LOCK_TTL_MS)]).catch((err: unknown) => {
logSafe("[oauth-lock] renew failed", err)
})
}, RENEW_EVERY_MS)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Refreshes outlive the lock

If Redis renewal fails for more than 15 seconds while a refresh is still running, the key expires and another instance can refresh the same token. The first holder keeps running: renewal errors only log, and a renewal result of 0 is ignored. The overlapping refreshes can replay a rotating token and revoke the session.

Bound the refresh work and handle lost ownership explicitly rather than continuing under an expired lock.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/auth/request-lock.ts
Line: 84-88

Comment:
**Refreshes outlive the lock**

If Redis renewal fails for more than 15 seconds while a refresh is still running, the key expires and another instance can refresh the same token. The first holder keeps running: renewal errors only log, and a renewal result of `0` is ignored. The overlapping refreshes can replay a rotating token and revoke the session.

Bound the refresh work and handle lost ownership explicitly rather than continuing under an expired lock.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

sessionStore: new RedisSessionStore(),
// Only one instance may refresh a DID's rotating refresh token at a
// time; a parallel refresh replays it and revokes the session.
requestLock: redisRequestLock,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Group timeouts delete valid sessions

Adding redisRequestLock introduces RequestLockTimeoutError into group requests, but getAuthenticatedAgent still calls deleteSession() after every restore error. If another instance holds the lock for 45 seconds, a request such as /api/groups/memberships deletes a valid app session and returns 401 instead of keeping it and returning 503.

Apply the shared error check to the group helper and preserve the temporary-failure response through its callers.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/auth/oauth-client.ts
Line: 122

Comment:
**Group timeouts delete valid sessions**

Adding `redisRequestLock` introduces `RequestLockTimeoutError` into group requests, but `getAuthenticatedAgent` still calls `deleteSession()` after every restore error. If another instance holds the lock for 45 seconds, a request such as `/api/groups/memberships` deletes a valid app session and returns 401 instead of keeping it and returning 503.

Apply the shared error check to the group helper and preserve the temporary-failure response through its callers.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

This branch was successfully deployed

1 active deployment
Preview — 55c538cb Deployed Oct 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants