POST /.well-known/workflow/v1/flow 500s with MessageNotAvailableError

[pensono]

I've been seeing this issue for a while across workflows v4 and v5 betas. I have an alert on any 500s which this issue is triggering. I believe this in benign, but I'd like to both confirm this and if possible have the framework handle it correctly. It seems that @vercel/queue@0.6.0 handles this case differently so perhaps upgrading is the best fix?

Report from my agent:

We've been seeing bursts of 500s on the generated workflow callback route:

Queue callback error: Error [MessageNotAvailableError]: Message <id> not available for processing

Most /.well-known/workflow/v1/flow requests succeed. These 500s show up in short spikes when a lot of callbacks fire at once (high fan-out, sometimes alongside 504s from long-running steps). The stack stays in the queue callback, not in our "use workflow" / "use step" code, and the runs themselves usually keep going.

From what we can tell, VQS delivers a callback, the consumer tries to claim the message by id, and VQS returns 409 because the message isn't available (already leased or otherwise not claimable). @vercel/queue@0.5.1 logs that as Queue callback error and responds 500. The polling client already treats the same error as { ok: false, reason: "not_available" } rather than a failure.

We're on workflow@5.0.0-beta.51 / @workflow/world-vercel@5.0.0-beta.46, which resolve @vercel/queue@0.5.1. Latest published Workflow packages still seem to sit on 0.5.1.

@vercel/queue@0.6.0 looks like it may address this: 409 / MessageNotAvailableError (and 410 / already processed) are documented as skipped with a 200 instead of a 500, so they wouldn't retry a message that can't be claimed. We haven't verified that in production. If that's the intended behavior, it might be worth pulling 0.6.0 into the Vercel World path when you're ready — otherwise we're happy to hear if this 500 is expected or if we're holding something incorrectly.

Related, but not the same error text: #925 / #1012 (visibility extension) and #579 (MessageNotFoundError).

[pranaygp]

(AI)

Severity: S4. Noise, not a correctness bug. Vercel Queues sometimes delivers a callback for a message that another consumer already holds (409, MessageNotAvailableError) or has already finished (410, MessageAlreadyProcessedError). This happens mostly under high fan-out or when deliveries overlap. The message still gets processed by whoever holds it, so runs aren't affected. But @vercel/queue@0.5.1's handleCallback logs Queue callback error and responds 500, which trips 5xx alerts. So yes, your read is right: these 500s are benign.

Reproduction. I drove the real @vercel/queue@0.5.1 callback path through @workflow/world-vercel's createQueueHandler, using a callback request with a stubbed 409 or 410 response to the claim-by-id call. The route returns 500 and never reaches the workflow handler. With @vercel/queue@0.6.0 it returns 200 { status: "skipped", reason: "not_available" | "already_processed" } and logs only at debug level. The only other change in 0.6.0 is an opt-in cursorFallback option for polling receives, which Workflow doesn't use.

Related issues.

• Queue callback error: Error [MessageNotFoundError]: Message id not found #579 is the same class of noise for 404 MessageNotFoundError. 0.6.0 does not change that case: it still returns 500, so Queue callback error: Error [MessageNotFoundError]: Message id not found #579 stays open.
• CPU-bound steps trigger phantom retries due to visibility timeout failure #925 / Workflow scaling issues #1012 cover visibility-timeout extension on long-running steps. They're a different mechanism, but they can cause the overlapping deliveries that end in these 409s.

Fix. #4333 bumps @vercel/queue to 0.6.0 and adds a regression test. Unit tests pass, and so do all Vercel prod E2E lanes. The one remaining red lane is a Next.js dev HMR-timing test that's unrelated to this change (the lane that was flaky on the previous run passed this time).

To see if this fix solves your issue, could you try to install the pre-release in your package.json? Just replace the workflow dep like so:

{
  "dependencies": {
    "workflow": "https://workflow-tarballs-6af0m3roo.labs.vercel.dev/workflow.tgz",
    "@workflow/world-vercel": "https://workflow-tarballs-6af0m3roo.labs.vercel.dev/workflow-world-vercel.tgz"
  }
}