Skip to content

fix(table): reclaim a cascade lock a timed-out acquire may have taken - #7680

Merged
waleedlatif1 merged 1 commit into
stagingfrom
fix/cascade-lock-reclaim
Sep 9, 2026
Merged

fix(table): reclaim a cascade lock a timed-out acquire may have taken#7680
waleedlatif1 merged 1 commit into
stagingfrom
fix/cascade-lock-reclaim

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • A client-side timeout does not mean Redis declined the SET. The command can still be parked in the offline queue and take the lock once the connection completes, leaving the row's cascade held for the full 30s TTL by an owner that already threw — no heartbeat, no release
  • Every other cell task for that row then reads contended and bails on the silent path, so one stalled connection quietly drops later cells rather than just failing the one run
  • Releasing after a failed acquire is what Redlock prescribes: a client that fails to acquire "will try to unlock all the instances (even the instances it believed it was not able to lock)", and the docs stress releasing ASAP so nobody waits out the TTL
  • Both preconditions the option documents hold here — ownerId is the cell task's unique executionId, and a throw means fn never runs, so the reclaim cannot cut under a caller still doing work

Verified against a local ioredis harness: at the moment the caller sees the timeout the server has only seen the handshake, and the SET arrives afterwards — so the lock really is taken by a caller that gave up. With the reclaim, the compare-and-delete lands behind it and frees the key.

The option stays opt-in rather than becoming the default: withLeaderLock and the MCP OAuth refresh mutex both return fn() when acquisition throws, so freeing a lock under them would admit a second concurrent runner.

Type of Change

  • Bug fix

Testing

Adds the cascade lock's first tests — acquire, contention, reclaim, release on throw, and heartbeat teardown. The reclaim test was verified to fail when the option is flipped off.

bun run type-check clean. 1,775 tests pass across lib/table and background. bun run lint, docs-manifest:check, and all 46 audits in check:audits pass.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

A client-side timeout does not mean Redis declined the SET. The command can
still be parked in the offline queue and take the lock once the connection
completes, leaving the row's cascade held for the full 30s TTL by an owner that
already threw — no heartbeat, no release. Every other cell task for that row
then reads `contended` and bails on the silent path, so one stalled connection
quietly drops later cells rather than just failing the one run.

Releasing after a failed acquire is what the Redlock algorithm prescribes:
a client that fails to acquire unlocks the instances anyway, including ones it
believed it had not locked. Both preconditions the option documents hold here —
`ownerId` is the cell task's unique `executionId`, and a throw means `fn` never
runs, so the reclaim cannot cut under a caller still doing work.

Adds the cascade lock's first tests, covering acquire, contention, reclaim,
release on throw, and heartbeat teardown.
@vercel

vercel Bot commented Sep 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 9, 2026 8:39pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR enables failed-acquisition reclamation for row cascade locks, preventing a Redis command that completes after a client-side timeout from silently holding the row lock until its TTL expires.

  • Opts only the cascade-lock acquisition path into reclaimOnFailure.
  • Documents why the cascade owner ID and execution flow make reclamation safe.
  • Adds coverage for acquisition, contention, failed-acquisition configuration, work failure, release, and heartbeat teardown.

Confidence Score: 5/5

The PR appears safe to merge; the reclamation path is owner-checked, limited to callers with unique execution IDs, and does not run concurrently with abandoned work.

No actionable failures remain: cleanup can only delete a lock carrying the failed caller’s unique owner ID, cleanup errors cannot replace the original acquisition error, and the relevant integration and lock-helper behavior are covered by tests.

Important Files Changed

Filename Overview
apps/sim/lib/table/cascade-lock.ts Safely opts cascade acquisition into owner-checked reclamation when Redis acquisition throws.
apps/sim/lib/table/cascade-lock.test.ts Adds focused tests for cascade-lock lifecycle behavior and verifies failed acquisitions request reclamation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Cell task requests cascade lock] --> B{Redis SET result}
    B -->|Acquired| C[Run cascade work]
    C --> D[Stop heartbeat and release owned lock]
    B -->|Contended| E[Return contended]
    B -->|Client-side error| F[Compare lock owner with execution ID]
    F -->|Matches| G[Delete potentially late-acquired lock]
    F -->|Does not match| H[Leave current holder untouched]
    G --> I[Rethrow original acquisition error]
    H --> I
Loading

Reviews (1): Last reviewed commit: "fix(table): reclaim a cascade lock a tim..." | Re-trigger Greptile

@waleedlatif1
waleedlatif1 merged commit 8ec065f into staging Sep 9, 2026
33 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/cascade-lock-reclaim branch September 9, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant