fix(execution): retry transient database failures during execution setup

[waleedlatif1]

Summary

• A dropped Postgres connection during workflow execution setup killed the run permanently. The first read in preprocessing is the workflow fetch; an ECONNRESET there surfaced as Internal error while fetching workflow, and background executions run with maxAttempts: 1, so there was no retry.
• Routed the read-only setup operations through the existing withDatabaseReadRetry helper (lib/db/read-retry.ts) instead of adding a second retry helper. Its classifier is the right one for reads — it excludes 57014 so a caller's own statement timeout still propagates.
• Covered every read-only operation on the setup path before the first block runs: 6 in preprocessing.ts, 5 in execution-core.ts (including getCustomBlockRowsForWorkspace, the first read after preprocessing returns).
• Safe because only reads are retried — the rate-limit token debit and the concurrency reservation are never re-entered. Retrying the whole preprocess would double-debit both.
• Added an optional label to withDatabaseReadRetry so the retry log line names which read flapped, matching the existing withTransactionRetry option shape.

Not wrapped, deliberately: getHighestPrioritySubscription swallows its own errors (onError: 'return-null') and never rejects, so a wrapper there would be dead code.

Type of Change

• Bug fix

Testing

Tested manually. bun run type-check, bun run lint, all 46 check:audits, and docs-manifest:check pass. 1,900 tests across lib/execution, lib/db, lib/workflows/executor, lib/knowledge/connectors and background pass.

Each new assertion was verified able to fail: unwrapping the call site turns the retry test red, and forcing the two parallel setup loads sequential turns the concurrency test red.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

🤖 Generated with Claude Code

https://claude.ai/code/session_01BYi7yz8qo98ziQWZmRpqb8

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 9, 2026 8:48pm UTC

[greptile-apps]

Greptile Summary

This PR makes workflow execution setup resilient to transient database connection failures by applying the existing bounded read-retry helper to read-only preprocessing and executor operations.

• Adds optional operation labels to retry warning logs.
• Retries workflow, billing, ban, usage, environment, custom-block, retention-policy, and run-identity reads.
• Preserves parallel workflow-state and environment loading.
• Adds retry coverage and improves the setup concurrency test.
• Avoids retrying admission mutations such as rate-limit debits and concurrency reservations.

Confidence Score: 5/5

The PR appears safe to merge; no outstanding correctness, security, or repository-rule issue was identified.

The retries are bounded to transient failures around read-only setup operations, preserve parallel loading, and do not repeat rate-limit or concurrency mutations. The previous type-safety thread was manually resolved after waleedlatif1 explained that the any casts had been replaced.

Important Files Changed

Filename	Overview
apps/sim/lib/db/read-retry.ts	Adds an optional diagnostic label while preserving the existing bounded transient-read retry behavior.
apps/sim/lib/execution/preprocessing.ts	Wraps six read-only preprocessing operations in database retries without re-entering admission mutations.
apps/sim/lib/execution/preprocessing.test.ts	Covers exhausted workflow-fetch retries and replaces prior any assertions in a manually resolved thread.
apps/sim/lib/workflows/executor/execution-core.ts	Retries executor setup reads while retaining concurrent workflow-state and environment loading.
apps/sim/lib/workflows/executor/execution-core.test.ts	Uses an event-loop boundary to assert setup-load concurrency without relying on a fixed microtask count.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Execution request] --> B[Preprocessing]
    B --> C[Retry read-only setup operations]
    C --> D[Rate-limit and concurrency admission]
    D --> E[Execution core]
    E --> F[Retry workflow state and environment loads]
    F --> G[Retry remaining setup reads]
    G --> H[Run first workflow block]
    C -. transient failure .-> C
    F -. transient failure .-> F
    G -. transient failure .-> G

Loading

_{Reviews (2): Last reviewed commit: "test(execution): type the logging-sessio..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.