feat(vector): support account-scoped embedding builds

[elviskahoro]

Add account and collection scoping to embedding builds, including durable [vector.embed.scope] accounts configuration for daemon jobs.

Scope is part of generation identity, preventing vectors from different account policies being mixed. Explicit empty collections fail closed, durable scope configuration resolves identifiers rather than unstable source IDs, and daemon jobs verify account-to-source mappings before embedding.

Coverage and generation reporting honor the resolved scope across SQLite and pgvector. The CLI documents repeatable --account and --collection flags.

[roborev-ci]

roborev: Combined Review (e9ec47a)

One medium-severity issue prevents valid out-of-scope identity coverage requests from returning zero results.

Medium

• internal/api/search_coverage.go:351 — A disjoint source scope is represented as SourceIDs: []int64{-1} while Identity.SourceID retains its original positive ID. The query validator rejects this mismatch, so a valid coverage request for an identity from an out-of-scope account errors instead of reporting zero eligible messages.
  • Fix: Preserve a valid identity/source pairing and represent the empty intersection using the existing impossible message-type sentinel, or clear/convert the identity predicate before applying the source sentinel. Add an endpoint test covering an identity belonging to an out-of-scope account.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 12m44s

[wesm]

looking

[roborev-ci]

roborev: Combined Review (0544377)

High-risk activation bug must be fixed; runtime scope changes also leave vector search incorrectly marked ready.

High

• internal/scheduler/embed_job.go:229 — The zero-live-message safeguard exists only in runEmbed, after the building generation has been created. On the next scheduled run, missing == 0 allows activation at line 247, retiring the working index and potentially deleting its pgvector embeddings. Default manual activation has the same gap.
  • Fix: Enforce the non-empty source-scope invariant in every non-forced activation path, preferably in the backend activation gate. Add scheduler and manual-activation coverage for source scopes with zero live messages.

Medium

• internal/scheduler/embed_job.go:153 — When a durable account resolves to a changed source-ID set, the scheduler only logs and skips embedding. The API and hybrid engine retain startup-time configuration, so searches validate against the old fingerprint and the vector subsystem remains “ready” instead of returning index_stale.
  • Fix: Propagate scope-resolution changes to API stale status, or re-resolve the expected scope during freshness checks so searches are blocked until vector features are reinitialized and rebuilt.

---

Reviewers: 2 done | Synthesis: codex, 11s | Total: 7m44s

[roborev-ci]

roborev: Combined Review (c3091d3)

Changes requested: two medium-severity gaps allow scope drift to go undetected or stale vector indexes to remain searchable.

Medium

• internal/api/vector_status.go:82 — SetVectorScopeDrift marks health as stale, but vector-search handlers only verify that the engine/backend exists. Hybrid, similar, and explore searches—including cached explore snapshots—can continue using an incorrectly scoped index when the installed fingerprint still matches.

  • Fix: Gate every vector-search entry point on VectorStatusStale, return index_stale, and add endpoint-level regression coverage.

• cmd/msgvault/cmd/serve_vector_init.go:181 — Scope drift is checked only when EmbedJob.Run executes. With an empty cron and run_after_sync=false, adding a linked source never triggers the check, so the expected index_stale transition does not occur.

  • Fix: Re-resolve scope independently of embedding scheduling, such as after source mutations/syncs or during vector-search preflight.

---

Reviewers: 2 done | Synthesis: codex, 13s | Total: 8m51s

[roborev-ci]

roborev: Combined Review (8350e3b)

Potential stale-index risk remains when deterministic account-scope resolution failures are treated as transient.

Medium

• internal/api/vector_status.go:161 — All scope-resolution errors are treated as transient. If an account is removed or becomes ambiguous, vector status remains ready and searches continue using cached source IDs. The scheduled job also skips without invoking OnScopeDrift at internal/scheduler/embed_job.go:156. On SQLite, reused message IDs could then hydrate old vectors as newly imported messages.
  • Fix: Distinguish transient database failures from deterministic invalid/not-found account errors. Mark deterministic failures as index_stale in both search preflight and scheduled-job paths, while continuing to retry transient failures.

---

Reviewers: 2 done | Synthesis: codex, 9s | Total: 15m58s

[roborev-ci]

roborev: Combined Review (03b3eb4)

Code is largely sound, but the coverage endpoint can report stale readiness when vector scope changes.

Medium

• internal/api/search_coverage.go:181 — The coverage endpoint refreshes an already-stale status but never calls maybeCheckVectorScopeDrift. With scheduling disabled, removing an account or changing linked sources can leave /api/v1/search/coverage reporting ready and calculating coverage from obsolete startup source IDs until a vector search occurs.
  • Fix: Run maybeCheckVectorScopeDrift before reading vector status in the coverage path, and add an endpoint test covering detected drift.

---

Reviewers: 2 done | Synthesis: codex, 11s | Total: 11m49s

[roborev-ci]

roborev: Combined Review (7c1ea8f)

No issues found.

---

Reviewers: 2 done | Synthesis: codex | Total: 13m0s

[wesm]

fixing the CI

[roborev-ci]

roborev: Combined Review (f08ab29)

Code is functionally sound, but one medium-severity UI integration issue needs fixing.

Medium

• internal/api/settings.go:113 — The API returns the new vector.embed.scope.accounts setting, but it is missing from web/src/lib/settings/catalog.ts. Because the web UI filters unknown catalog keys, users cannot view or edit this setting. Add it to the web catalog and cover it in the catalog test.

---

Reviewers: 2 done | Synthesis: codex, 14s | Total: 12m31s

[roborev-ci]

roborev: Combined Review (f884637)

The changes are generally sound, but two medium-severity stale-scope and generation-drift issues remain.

Medium

• Reused source IDs can bypass scope drift detection — cmd/msgvault/cmd/serve_vector_init.go:199
  Drift detection compares only resolved source-ID fingerprints. Because SQLite may reuse the highest deleted INTEGER PRIMARY KEY, a newly created account can appear unchanged and the cached worker may embed unintended content across the documented privacy boundary. Track a non-reusable source incarnation or stable identity in the scope and generation fingerprint, or ensure source IDs are never reused.

• Cached Explore snapshots can serve a previous generation — internal/api/vector_status.go:179
  Preflight checks only cached stale status and does not verify that the active generation matches vectorCfg.GenerationFingerprint(). After a one-off scoped build activates through the daemon subprocess, the parent can remain ready, allowing cached snapshots from the previous generation instead of returning index_stale. Resolve the active generation during preflight and set a clearable stale status when ErrIndexStale occurs.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 9m0s

[roborev-ci]

roborev: Combined Review (3599f43)

No issues found.

---

Reviewers: 2 done | Synthesis: codex | Total: 11m6s

[roborev-ci]

roborev: Combined Review (c63be5d)

Review verdict: One medium-severity issue should be addressed before merge.

Medium

• Stale Explore snapshots can bypass generation fingerprint validation — internal/api/explore.go:1423

  After a daemon-fronted one-off --account/--collection build activates, the daemon can remain ready because its configured scope has not drifted. While ready, preflight does not compare the active generation’s fingerprint, allowing cached Explore snapshots to serve candidates from the previous generation instead of returning index_stale.

  Suggested fix: Validate the active generation against the expected fingerprint before reusing a snapshot, or perform this validation in vectorSearchPreflight and mark the server stale on mismatch.

---

Reviewers: 2 done | Synthesis: codex, 17s | Total: 16m28s

[roborev-ci]

roborev: Combined Review (bb730c1)

Code review found two medium-severity issues involving stale vector status and durable scope identity.

Medium

• internal/api/vector_status.go:111 — After a daemon-proxied one-off scoped build activates, the installed configuration retains the old fingerprint. Because refreshVectorStatusIfStale validates only already-stale statuses, health and stats continue reporting ready while vector searches return index_stale.

  • Fix: Periodically revalidate ready states, or notify the server after generation activation and mark the status stale when the active fingerprint differs.

• cmd/msgvault/cmd/embed_scope.go:92 — Durable scope configuration accepts display names, but drift detection compares only resolved numeric source IDs. If a source is removed and SQLite reuses its ID for another source with the same display name, the replacement account’s content could be sent to the embedding endpoint, violating the privacy boundary.

  • Fix: Require canonical identifiers for durable configuration, or persist and compare each source’s stable type/identifier identity alongside its numeric ID.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 9m9s

[roborev-ci]

roborev: Combined Review (e99e36b)

Account-scoped embedding is generally sound, but one medium-severity resolution bug can silently omit mailbox data.

Medium

• Case-sensitive primary email resolution can exclude the mailbox — cmd/msgvault/cmd/embed_scope.go:102

  Primary email sources are resolved case-sensitively, while linked Calendar sources are resolved case-insensitively. A stored Gmail identifier such as User@Acme.com, queried as user@acme.com, can resolve only to Calendar sources and silently omit the mailbox from the embedding build.

  Fix: Resolve email-shaped primary identifiers case-insensitively before accepting the Calendar-only fallback, then union the primary and linked source IDs.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 12m58s

[roborev-ci]

roborev: Combined Review (4561505)

Code is functionally sound, but activation ordering introduces a medium-severity database performance risk.

Medium

• internal/vector/pgvector/backend.go:359 — The active generation is demoted and its embeddings deleted before checking whether the new scope is empty. Although rollback preserves the data, each scheduled activation attempt for an empty scope can trigger a corpus-sized DELETE and rollback, causing excessive locking, WAL generation, and database load. Check for at least one live in-scope message before demoting or deleting the active generation, while retaining the gated promotion check to handle races.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 14m13s

[roborev-ci]

roborev: Combined Review (58ad251)

Code is generally sound, but one medium-severity information disclosure should be fixed before merge.

Medium

• cmd/msgvault/cmd/serve_vector.go:97 — Account-resolution errors are passed verbatim to vector initialization status and exposed through the unauthenticated /health endpoint. Missing or ambiguous account errors may reveal mailbox/service identifiers, including matching accounts; runtime drift errors can leak the same information through embedScopeDriftCheck. Log detailed resolver errors and expose them only through authenticated /api/v1/health, while returning a generic vector error/status from public /health.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 14m59s

[roborev-ci]

roborev: Combined Review (cd7688b)

Changes look sound overall, but one medium-severity readiness gap remains.

Medium

• internal/api/server.go:1228, internal/api/server.go:1241, internal/api/handlers.go:552 — Health and stats handlers refresh fingerprint freshness but do not run the new account-scope drift check. When embedding scheduling is disabled, removing or remapping a configured account can leave these endpoints reporting ready until vector search or coverage triggers the check. Call maybeCheckVectorScopeDrift before reading vector status in both health handlers and handleStats, using the existing throttle.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 18m20s

[roborev-ci]

roborev: Combined Review (977aa5e)

Contextual account scoping is unsafe, and two activation/convergence paths can mishandle account-scoped generations.

High

• Account scope is bypassed by contextual embedding paths — internal/vector/embed/context_worker.go:92, internal/vector/embed/context_worker.go:652-665, internal/vector/embed/context_worker.go:1428-1510, internal/vector/embed/context_worker.go:1646-1767, internal/vector/embed/context_worker.go:1882-1894

  SourceIDs are preserved and used during initial discovery, but downstream selector validation, reconciliation, journal processing, metadata fan-out, and final source assembly enforce only message type. Source-only scopes can be rejected incorrectly, while combined source/type scopes can send excluded-account content to the external embedding provider and publish out-of-scope vectors.

  Enforce source and message-type constraints independently throughout contextual processing, include source identity in routing metadata, and add tests proving excluded-account text never reaches the embedding client.

Medium

• Empty contextual generations can be activated — internal/vector/sqlitevec/backend.go:497

  ActivateGenerationIfConverged lacks the empty-source-scope guard, allowing a zero-message contextual generation to demote the serving generation and activate an empty index.

  Check for a live in-scope message within the fused transaction before demotion and return ErrRefuseActivateEmptyScope when none exists.

• Contextual convergence uses unresolved account scope — cmd/msgvault/cmd/embeddings_manage.go:550

  The activation plan resolves account IDs into vecCfg, but configuredConvergenceState builds its checker from unresolved global cfg.Vector. Out-of-scope messages may therefore be counted as missing, preventing activation of a completed account-scoped generation.

  Pass the resolved vecCfg through requireConfiguredConvergence and configuredConvergenceState, and use it to construct the convergence checker.

---

Reviewers: 2 done | Synthesis: codex, 15s | Total: 9m29s

[roborev-ci]

roborev: Combined Review (11f620a)

Two medium-severity issues remain in embedding scope identity and configuration reproducibility.

Medium

• cmd/msgvault/cmd/embed_scope.go:65 — Collection scopes may not be reproducible by durable configuration.
  --collection fingerprints exact source IDs, while durable configuration resolves entries using account semantics that may expand linked sources or reject duplicate identifiers. A valid collection can therefore activate a generation that no [vector.embed.scope].accounts value can reproduce, preventing the daemon from serving it without rebuilding.
  Fix: Support durable collection/exact-source selectors, or reject collection scopes that cannot be represented by canonical account configuration before starting the build.

• cmd/msgvault/cmd/embed_scope.go:139 — Reused SQLite source IDs can bypass drift detection.
  Drift detection compares only resolved source IDs, but sources.id INTEGER PRIMARY KEY can reuse the highest deleted ID. Deleting a scoped source and creating a replacement with the same ID may preserve the scope fingerprint, allowing replacement-source text to flow through components initialized for the deleted source.
  Fix: Include an immutable source-incarnation identifier in the fingerprint, or guarantee non-reused source identities and add a delete/recreate regression test.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 11m55s

[roborev-ci]

roborev: Combined Review (b1bac6e)

No issues found.

---

Reviewers: 2 done | Synthesis: codex | Total: 14m59s