Track Vercel request IDs on workflow events for observability#997
Draft
Track Vercel request IDs on workflow events for observability#997
Conversation
Extract the x-vercel-id header from incoming requests in queue handlers and propagate it through event creation so every workflow event records which Vercel request created it. This enables linking Vercel request logs with workflow runs and step executions in observability. Changes: - Add requestId to Queue handler metadata and CreateEventParams - Add requestId field to Event schema - Extract x-vercel-id in Vercel queue handler (via AsyncLocalStorage) and local queue handler - Pass requestId through workflow runtime, step handler, and suspension handler to all event creation calls - Store requestId in world-vercel, world-local, and world-postgres event storage - Accept optional requestId in StartOptions for initial workflow start https://claude.ai/code/session_01TqomNcQoixA1HzxvFFBsu2
🦋 Changeset detectedLatest commit: bd96bc3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 18 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests💻 Local Development (41 failed)nuxt-stable (41 failed):
🌍 Community Worlds (41 failed)turso (41 failed):
Details by Category✅ ▲ Vercel Production
❌ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end propagation and persistence of Vercel request IDs (x-vercel-id) so workflow/step events can be correlated with request-level logs in observability tooling.
Changes:
- Extends event and queue metadata types to carry an optional
requestId, and threads it through core runtime event creation calls. - Extracts
x-vercel-idin Vercel and local queue handlers and propagates it into queue callback metadata (viaAsyncLocalStoragefor@vercel/queue). - Persists
requestIdon events in Postgres and local file event storage, and forwards it through the Vercel API client.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/world/src/queue.ts | Adds requestId?: string to queue handler metadata type. |
| packages/world/src/events.ts | Adds requestId?: string to CreateEventParams and to EventSchema. |
| packages/world-vercel/src/queue.ts | Captures x-vercel-id from incoming requests and propagates via AsyncLocalStorage. |
| packages/world-vercel/src/events.ts | Forwards requestId in v2 event creation requests. |
| packages/world-postgres/src/storage.ts | Writes requestId into event inserts and includes it in returned event objects. |
| packages/world-postgres/src/drizzle/schema.ts | Adds request_id column to the Drizzle events table schema. |
| packages/world-local/src/storage/events-storage.ts | Includes requestId when writing local event JSON. |
| packages/world-local/src/queue.ts | Extracts x-vercel-id and passes it into handler metadata. |
| packages/core/src/runtime/suspension-handler.ts | Propagates requestId into suspension-created events. |
| packages/core/src/runtime/step-handler.ts | Propagates requestId into step lifecycle events. |
| packages/core/src/runtime/start.ts | Adds StartOptions.requestId and passes it into run_created event creation. |
| packages/core/src/runtime.ts | Propagates requestId into workflow lifecycle events and suspension handling. |
| .changeset/request-id-tracking.md | Declares patch releases for the affected packages. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
106
to
+108
| specVersion: integer('spec_version'), | ||
| /** The Vercel request ID (x-vercel-id header) that created this event */ | ||
| requestId: varchar('request_id'), |
There was a problem hiding this comment.
A Drizzle migration needs to be added for this new request_id column. The package’s db:push script runs drizzle-orm migrations (see src/cli.ts), so without a new migrations/*.sql entry that ALTERs workflow.workflow_events to add request_id, installs/upgrades will fail at runtime when inserts include requestId.
packages/world-vercel/src/queue.ts
Outdated
| // Wrap the VQS handler to extract x-vercel-id from the incoming request | ||
| // and propagate it via AsyncLocalStorage into the handler callback | ||
| return async (req: Request) => { | ||
| const requestId = req.headers.get(VERCEL_REQUEST_ID_HEADER) ?? undefined; |
There was a problem hiding this comment.
Header extraction uses req.headers.get(...) ?? undefined, which preserves an empty header value as ''. That then propagates as a (falsy) requestId through metadata and can lead to inconsistent persistence/serialization downstream. Consider normalizing here (e.g., trim and convert '' to undefined) so the rest of the pipeline can treat “missing/blank” consistently.
Suggested change
| const requestId = req.headers.get(VERCEL_REQUEST_ID_HEADER) ?? undefined; | |
| const rawRequestId = req.headers.get(VERCEL_REQUEST_ID_HEADER); | |
| const requestId = rawRequestId?.trim() || undefined; |
Comment on lines
193
to
207
| // Extract x-vercel-id for request ID tracking (available on Vercel deployments) | ||
| const requestId = req.headers.get('x-vercel-id') ?? undefined; | ||
|
|
||
| if (!queueName.startsWith(prefix)) { | ||
| return Response.json({ error: 'Unhandled queue' }, { status: 400 }); | ||
| } | ||
|
|
||
| const body = await new JsonTransport().deserialize(req.body); | ||
| try { | ||
| const result = await handler(body, { attempt, queueName, messageId }); | ||
| const result = await handler(body, { | ||
| attempt, | ||
| queueName, | ||
| messageId, | ||
| requestId, | ||
| }); |
There was a problem hiding this comment.
Header extraction uses req.headers.get('x-vercel-id') ?? undefined, which keeps an empty header value as ''. Since downstream code often gates on truthiness, this can result in the requestId being dropped from emitted events even though it was “present”. Consider normalizing (trim + convert blank to undefined) before passing it into the handler metadata.
Comment on lines
393
to
397
| eventType: data.eventType, | ||
| eventData: 'eventData' in data ? data.eventData : undefined, | ||
| specVersion: effectiveSpecVersion, | ||
| requestId: params?.requestId ?? null, | ||
| }) |
There was a problem hiding this comment.
This change adds new persisted/returned behavior (requestId stored on events). There are extensive integration tests in packages/world-postgres/test/storage.test.ts, but none currently assert requestId round-trips. Adding at least one test that calls events.create(..., ..., { requestId }) and verifies the returned event (and/or a subsequent list/get) includes the same requestId would prevent regressions.
Comment on lines
396
to
406
| requestId: params?.requestId ?? null, | ||
| }) | ||
| .returning({ createdAt: Schema.events.createdAt }); | ||
|
|
||
| const result = { ...data, ...value, runId: effectiveRunId, eventId }; | ||
| const result = { | ||
| ...data, | ||
| ...value, | ||
| runId: effectiveRunId, | ||
| eventId, | ||
| ...(params?.requestId ? { requestId: params.requestId } : {}), | ||
| }; |
There was a problem hiding this comment.
params?.requestId is written to the DB using ?? null, but the returned event object only includes requestId when it’s truthy (params?.requestId ? ...). This can create inconsistencies (e.g., an empty-string requestId is persisted but omitted from the returned event) and makes behavior dependent on truthiness. Consider normalizing requestId (treat blank as undefined) and using an explicit undefined/null check when deciding whether to include it in the result.
Comment on lines
858
to
878
| @@ -866,6 +874,7 @@ export function createEventsStorage(drizzle: Drizzle): Storage['events'] { | |||
| ...conflictValue, | |||
| runId: effectiveRunId, | |||
| eventId, | |||
| ...(params?.requestId ? { requestId: params.requestId } : {}), | |||
| }; | |||
There was a problem hiding this comment.
Same truthiness issue as other result constructions: requestId is persisted via ?? null, but then only included in the returned conflictResult when params?.requestId is truthy. Prefer an explicit undefined/null check (or normalize blank to undefined earlier) so storage and returned event stay consistent.
| ...value, | ||
| runId: effectiveRunId, | ||
| eventId, | ||
| ...(params?.requestId ? { requestId: params.requestId } : {}), |
There was a problem hiding this comment.
requestId is always inserted (as null when missing), but the returned result only includes it when truthy. This can drop valid-but-falsy values (e.g., empty string) and makes behavior inconsistent across code paths. Consider normalizing the input and switching to an explicit undefined/null check when adding requestId to the returned event object.
Suggested change
| ...(params?.requestId ? { requestId: params.requestId } : {}), | |
| ...(params?.requestId !== undefined | |
| ? { requestId: params.requestId ?? null } | |
| : {}), |
Comment on lines
136
to
149
| const result = await world.events.create( | ||
| runId, | ||
| { | ||
| eventType: 'run_created', | ||
| specVersion, | ||
| eventData: { | ||
| deploymentId: deploymentId, | ||
| workflowName: workflowName, | ||
| input: workflowArguments, | ||
| executionContext: { traceCarrier, workflowCoreVersion }, | ||
| }, | ||
| }, | ||
| { v1Compat } | ||
| { v1Compat, requestId: opts?.requestId } | ||
| ); |
There was a problem hiding this comment.
StartOptions adds requestId and threads it into world.events.create(...). There are existing unit tests for start() in packages/core/src/runtime/start.test.ts, but none currently assert that options are forwarded into the create-event params. Adding a test that passes requestId and verifies the mocked world.events.create receives it would lock in this behavior.
| eventId, | ||
| createdAt: now, | ||
| specVersion: effectiveSpecVersion, | ||
| ...(params?.requestId ? { requestId: params.requestId } : {}), |
There was a problem hiding this comment.
requestId is added to the stored event only when it’s truthy. If an empty-string requestId slips through (e.g., a present-but-blank header), it will be silently dropped, making behavior differ from “explicitly provided but blank”. Consider normalizing requestId earlier (trim + convert blank to undefined) and/or switching this condition to an explicit undefined/null check for consistency with other storages.
Suggested change
| ...(params?.requestId ? { requestId: params.requestId } : {}), | |
| ...(params?.requestId !== undefined && params?.requestId !== null | |
| ? { requestId: params.requestId } | |
| : {}), |
- Add requestId to attributeToDisplayFn in web-shared (fixes TS2741 build error) - Add Drizzle migration 0007 for request_id column on workflow_events - Normalize empty-string requestId values in queue handlers (trim + falsy check) - Use consistent || null for requestId persistence in postgres storage - Include @workflow/web-shared in changeset https://claude.ai/code/session_01TqomNcQoixA1HzxvFFBsu2
Collaborator
Author
|
gonna spend a bit more time thinking through this flow properly before re-opening it out of draft |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for tracking Vercel request IDs (
x-vercel-idheader) throughout the workflow execution lifecycle. Request IDs are now captured from incoming HTTP requests and propagated to all workflow and step events, enabling better correlation between request logs and workflow executions in observability systems.Backend PR needed for store these IDs: https://github.com/vercel/workflow-server/pull/251
Key Changes
requestIdparameter to theCreateEventParamsinterface and threaded it through all event creation calls in the workflow runtime, step handler, and suspension handler@vercel/queuehandler wrapper to extract thex-vercel-idheader from incoming requests usingAsyncLocalStoragerequestIdto the queue handler metadata interfacerequestIdfield on eventsrequestIdin event creation requestsrequestIdfield toEventSchemaand documented it in relevant interfacesrequestIdcolumn to PostgreSQL events tableImplementation Details
AsyncLocalStorageto propagate request IDs through the async call chain in the Vercel queue handler, since the@vercel/queuelibrary abstracts away the raw Request objecthttps://claude.ai/code/session_01TqomNcQoixA1HzxvFFBsu2