fix(knowledge): make indexing and connector recovery durable

[icecrasher321]

Summary

• Coordinate OCR and connector provider quotas across workers, honor cooldowns, and preserve successful work through delayed retries and bounded processing slices.
• Checkpoint verified OCR ranges and embedding batches while keeping complete-document indexing atomic. Package the shared PDF runtime for Linux and report permanent input failures without repeated provider calls.
• Persist backing-file cleanup before uploads and inside deletion transactions, protect concurrent re-uploads, and account for retained source documents when disconnecting connectors.
• Fix empty Confluence pages, oversized skipped-source metadata, and strict connector-detail response projection.
• Apply the additive capacity-state migration before rolling out app and worker changes. Included runbooks cover quota tuning, historical storage reconciliation, and reviewed orphan cleanup.

Type of Change

• Bug fix

Testing

• 2,619 focused unit tests passed; one conditional test skipped.
• 226 integration checks passed with disposable PostgreSQL/Redis and real storage, covering failure injection, checkpoint resumption, atomic indexing, accounting and authorized search. The integration run skipped 42 conditional cases. Live OpenAI/Mistral calls use synthetic content; optional live connector-account suites require separate fixtures.
• Linux Next.js standalone PDF runtime passed on ARM64 and AMD64, including concurrent cold-start and warm requests with network disabled.
• Helm lint, 108 chart tests, scheduler parity and image inventory checks passed.
• Lint, full application TypeScript, all 46 repository audits (including strict API validation), migration safety, block registry and generated-artifact checks passed.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 8, 2026 11:36pm UTC

[icecrasher321]

@cubic review

[cubic-dev-ai]

@cubic review

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[icecrasher321]

@greptile-apps

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the previously reported connector-cleanup race is resolved and no new actionable regression remains.

Summary

• Coordinates provider admission, cooldowns, and durable processing continuations.
• Checkpoints OCR and embedding progress while preserving atomic completed-document indexing.
• Protects connector uploads through attachment and persists storage cleanup/accounting.
• Adds connector edge-case handling, deployment configuration, migration state, tests, and operational documentation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Document or connector source] --> B[Reserve metadata and cleanup guard]
    B --> C[Upload source artifact]
    C --> D[Lock cleanup guard]
    D --> E[Attach artifact in transaction]
    E --> F[Process OCR and embeddings]
    F --> G{Provider capacity available?}
    G -- No --> H[Persist checkpoint and schedule continuation]
    H --> F
    G -- Yes --> I[Checkpoint successful batches]
    I --> J[Atomically publish completed index]
    E -. rollback .-> K[Cleanup guard remains eligible]
    K --> L[Delete unreferenced artifact]

Loading

[icecrasher321]

@greptile-apps

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.