[Feature] Improve embedder model migration experience

[A0nameless0man]

Problem

Switching the embedding model in OpenViking requires manual, error-prone steps with no validation and no batch tooling. The process overwrites vectors in-place, causing search quality degradation visible to end-users during the migration window. A blue-green migration strategy would eliminate user-visible impact -- search always hits a complete, consistent vector set.

Current Workflow

# 1. Admin manually edits ov.conf -- no validation at all
vim ~/.openviking/ov.conf
#   modify: embedding.dense.model, dimension, api_base...

# 2. Restart the server -- downtime, no pre-check
docker restart openviking
# or:
openviking-server  # starts without any verification

# 3. Reindex resources one by one -- no batch operation
ov reindex viking://resources/doc-a --regenerate --wait
ov reindex viking://resources/doc-b --regenerate --wait
# ... repeat for every resource

# 4. User-visible impact:
#   - Search quality degrades silently (old vectors + new model = mismatch)
#   - No notification that embedding has changed
#   - No way to confirm reindex completion

Pain Points

Admin

Problem	Impact	Evidence
No endpoint-level dimension validation	Config layer has a VectorDB vs Embedding dimension consistency WARNING (openviking_cli/utils/config/open_viking_config.py) and auto-syncs -- but it never verifies whether the configured dimension matches the embedding endpoint's actual output	openviking-server doctor only checks if API key is set, does not test endpoint connectivity
No bulk reindex	Must run ov reindex <uri> individually for each resource	Current reindex endpoint accepts only a single URI
No model compatibility check	Config-time model/provider compatibility is not validated. Errors surface only at query time. Example: provider=openai + a downstream model that doesn't support matryoshka representations (e.g. certain OpenAI-compatible Qwen endpoints) -- v0.3.5 passed through the dimensions parameter causing a 400 error (#1442, fixed in v0.3.6). Or provider=litellm + bare model name (Qwen3-Embedding-0.6B instead of dashscope/Qwen3-Embedding-0.6B) -- query fails with LLM Provider NOT provided	#1442 -- users only discover config errors at search time
No progress visibility	ov reindex --wait blocks with no progress indication	No progress events
No rollback path	After reindexing with a new model, old vectors are gone	build_index() overwrites in-place

User

Problem	Impact
Search quality degradation during migration	reindex overwrites vectors in-place. During the migration window, query vectors use the new model while some data still has old model vectors -- results are unpredictable
No atomicity	No all-or-nothing switchover. Users may hit a half-old-half-new vector set

Users should always hit a complete, consistent vector set during migration. Blue-green migration (build new vector set in background, then atomic active-pointer switchover) is the straightforward solution.

Proposed Solution

Dependency on #1439: This proposal builds on #1439 (feat: detect embedding model drift and add rebuild tool), which provides:

• embedding_compat.py -- embedding identity persistence (embedding_meta.json), compatibility_identity(), and startup-time ensure_embedding_collection_compatibility() check
• vector_rebuild.py -- VectorRebuildService (discover_accounts(), rebuild_account(), rebuild_accounts())
• openviking-rebuild-vectors CLI -- --all-accounts batch rebuild entry point
• EmbeddingCompatibilityError -- startup fail-fast mechanism

This proposal adds blue-green migration, dual-write, and atomic switchover on top of #1439. If #1439 is not yet merged, Section 4 (health gate) and Section 6 (migration resilience) will need the embedding_meta.json persistence part implemented first.

1. Extend ov reindex with batch support

ov reindex <URI> currently accepts only a single URI. When switching embedding models, admins need to reindex every resource -- doing this one at a time is impractical.

# Current:
ov reindex viking://resources/doc-a --regenerate --wait

# Proposed:

# Reindex all resources
ov reindex --all --regenerate --wait=false

# Reindex with glob pattern
ov reindex viking://resources/my-project/** --regenerate

# Dry-run: show what would be reindexed (count + estimated time)
ov reindex --all --dry-run
# Output:
#   Resources to reindex: 47
#   Estimated time: ~12 min
#   Current embedding model: text-embedding-v4 (1024d)
#   WARNING: Dimension mismatch detected -- vectordb rebuild may be required

2. Extend ov config validate with live endpoint check

openviking-server doctor (openviking_cli/doctor.py) already checks config syntax, Python version, native engine, AGFS, embedding API key existence, VLM config, and disk space. It does not test endpoint connectivity or verify actual embedding dimensions. This proposal adds those checks either to doctor or to a new ov config validate --live command. The two don't conflict -- doctor covers operational health, --live covers pre-change validation.

ov config validate currently only checks config syntax (JSON schema via serde). Extend it to verify the endpoint is reachable and the output dimension matches config.

# Current:
ov config validate
# -> only checks JSON schema

# Proposed:
ov config validate --live
# Checks:
#   PASS: Config syntax valid
#   PASS: Embedding endpoint reachable
#   PASS: Embedding dimension matches config (1024d)
#   WARNING: Embedding model differs from stored model (text-embedding-v3 -> text-embedding-v4)
#   Note: Run `ov reindex --all` to rebuild vectors

2.1 Config structure for blue-green

During migration, the system needs both the current (active) model config and the target model config. We propose a named embedding.migration map in ov.conf, where each entry is a named migration target. The existing embedding config is implicitly named default.

embedding.migration is a map of named configs, not a single block. This design:

1. Supports read-only ov.conf -- migration targets are pre-defined, no runtime writes needed (works with container read-only mounts)
2. Allows multiple migration targets -- admins can pre-configure several models and pick one via CLI
3. Makes default implicit -- the existing top-level embedding config (dense/sparse/hybrid) is the active profile

Background: EmbeddingConfig supports three embedding types:

• dense -- dense vectors (most common)
• sparse -- sparse vectors (BM25-style)
• hybrid -- single model returning both dense + sparse

get_embedder() logic: if hybrid exists, use hybrid embedder; if both dense and sparse exist, use CompositeHybridEmbedder; if only dense, use dense embedder.

Migration configs need to cover all three cases. Each migration entry mirrors the model config fields in embedding:

Case A: dense only (most common)

{
  "embedding": {
    "dense": {
      "provider": "volcengine",
      "model": "doubao-embedding-vision-251215",
      "dimension": 1024,
      "api_base": "https://ark.cn-beijing.volces.com/api/v3",
      "api_key": "..."
    },
    "migration": {
      "openai-v3-large": {
        "dense": {
          "provider": "openai",
          "model": "text-embedding-3-large",
          "dimension": 3072,
          "api_base": "https://api.openai.com/v1",
          "api_key": "..."
        }
      }
    },
    "max_concurrent": 10
  }
}

Case B: dense + sparse (composite hybrid)

{
  "embedding": {
    "dense": { "provider": "volcengine", "model": "...", "dimension": 1024 },
    "sparse": { "provider": "volcengine", "model": "..." },
    "migration": {
      "openai-mixed": {
        "dense": { "provider": "openai", "model": "...", "dimension": 3072 },
        "sparse": { "provider": "openai", "model": "..." }
      }
    },
    "max_concurrent": 10
  }
}

Case C: hybrid (single-model hybrid)

{
  "embedding": {
    "hybrid": { "provider": "volcengine", "model": "...", "dimension": 1024 },
    "migration": {
      "openai-hybrid": {
        "hybrid": { "provider": "openai", "model": "...", "dimension": 3072 }
      }
    },
    "max_concurrent": 10
  }
}

Note: Each migration entry mirrors only dense/sparse/hybrid model config fields. Top-level runtime settings (max_concurrent, circuit_breaker, max_retries, etc.) are global and don't change during migration.

CLI references migration targets by name:

# List available migration targets
ov reindex --list-targets
# Output:
#   Available migration targets:
#   - openai-v3-large  (openai/text-embedding-3-large, 3072d)
#   - qwen-3-large     (dashscope/qwen3-embedding, 1024d)

# Start migration with a pre-configured target
ov reindex --all --target openai-v3-large

Lifecycle:

Phase	Active profile	Migration state	Behavior
Normal	default	None	Single-model operation
Migration start	default	Target name selected via CLI	Dual-write + bulk re-embed to target
Migration complete	Auto-switched to target	Target entry removed from config	Single-model, new config becomes default
Rollback	Reverted to default	Target re-added	Dual-write back to old

3. Blue-green vector migration

Instead of overwriting vectors in-place during reindex, maintain two vector sets ("blue" = current active, "green" = new model being built). Users always query the active set. Once the green set is fully built and verified, atomically promote it to active.

Changing model and changing dimension are the same operation

With in-place overwrite, changing the embedding dimension (e.g. 1024d to 3072d) requires dropping and recreating the entire vectordb -- destructive and irreversible. With blue-green, both "new model" and "new dimension" are handled the same way: write to the inactive collection, then flip the pointer. No schema migration, no data loss, no downtime.

Migration timeline

flowchart TD
    A["Admin starts reindex with new model"] --> B["Phase 1: Enable dual-write<br/>New writes -> default + openai-v3-large<br/>Queries -> default"]
    B --> C["Phase 2: Bulk re-embed existing resources<br/>default (active): text-embedding-v3, 47 resources<br/>openai-v3-large (building): 12/47...<br/>Queries -> default<br/>Dual-write active"]
    C -->|"openai-v3-large complete"| D["Phase 3: Query switchover<br/>Active pointer: default -> openai-v3-large<br/>Queries -> openai-v3-large<br/>Dual-write still active"]
    D --> E["Phase 4: Disable dual-write<br/>Writes -> openai-v3-large only<br/>default retained for rollback until TTL expires"]
    E -->|"TTL expires or admin confirms"| F["Phase 5: Delete old set<br/>Delete default collection<br/>openai-v3-large becomes the new default"]

Loading

API surface

class VectorDB:
    def get_active_collection(self) -> str:
        """Returns collection name for reads -- e.g. 'default' or 'openai-v3-large'."""
        return self.metadata.get("active_embedding_set", "default")

    def is_dual_write_enabled(self) -> bool:
        return self.metadata.get("dual_write", False)

    def switch_active(self, target: str) -> None:
        """Atomic metadata write -- instant switchover for all reads."""
        self.metadata.set("active_embedding_set", target)

    def set_dual_write(self, enabled: bool) -> None:
        self.metadata.set("dual_write", enabled)

    def upsert(self, resource_uri: str, vector) -> None:
        """In dual-write mode, writes to both active and inactive collections."""
        active = self.get_active_collection()
        self._write_to_collection(active, resource_uri, vector)
        if self.is_dual_write_enabled():
            inactive = self._get_inactive_collection()
            self._write_to_collection(inactive, resource_uri, vector)

The migration controller orchestrates phases through these primitives:

1. set_dual_write(True) -- enable dual-write
2. Loop: embed_with_new_model() then _write_to_collection(green, ...) -- bulk re-embed to green only
3. switch_active(green) -- atomic query switchover
4. set_dual_write(False) -- disable dual-write
5. delete_collection(blue) -- cleanup old set

Rollback

# If admin detects quality regression after switchover:
ov reindex --rollback
# Instantly switches back to previous set (still on disk)

# Rollback behavior varies by phase:
#   Phase 1-2 (dual-write/building): disable dual-write, discard green set
#   Phase 3-4 (switched/dual-write-off): flip pointer back to blue, re-enable dual-write briefly
#   Phase 5 (cleanup): too late, blue already deleted -- full reindex required

4. Runtime embedding model health gate

On server startup, validate embedding config against existing vectordb state:

# NOTE: Current vectordb metadata does not store embedding_model name (only dimension).
# This pseudocode assumes #1066/#1439's embedding identity metadata is implemented.
current_model = config.embedding.dense.model
stored_model = vectordb.get_metadata("embedding_model")  # requires #1066/#1439
stored_dim = vectordb.get_metadata("embedding_dimension")

if current_model != stored_model or config.embedding.dense.dimension != stored_dim:
    log.warning(
        f"Embedding model changed: {stored_model} ({stored_dim}d) -> {current_model} ({config.embedding.dense.dimension}d). "
        f"Run 'ov reindex --all' to rebuild vectors with blue-green migration."
    )

5. Config validation on load

When ov.conf is loaded, immediately test the embedding endpoint with a trivial input (not on first search):

# Validate on config load, not on first search
try:
    result = await embedder.embed("health check", is_query=True)
    assert len(result.dense_vector) == config.embedding.dense.dimension
except Exception as e:
    raise ConfigError(f"Embedding endpoint validation failed: {e}")

6. Migration resilience

Migration can take hours. If the server restarts or the CLI disconnects, all progress is lost without persistent state.

Building on #1439: #1439 provides embedding identity persistence (embedding_meta.json) and VectorRebuildService (per-account delete + reindex). This section extends that foundation:

• Migration state -- feat: detect embedding model drift and add rebuild tool #1439's embedding_meta.json records the active embedding identity; migration_state.json (this proposal) records blue-green switchover progress and phase. They complement each other, no overlap.
• Bulk re-embed -- feat: detect embedding model drift and add rebuild tool #1439's VectorRebuildService.rebuild_account() does in-place overwrite (delete + rebuild from AGFS). Blue-green replaces that with "write to green set + atomic switchover", but reuses discover_accounts() and _discover_directories() traversal logic.
• Startup check -- feat: detect embedding model drift and add rebuild tool #1439's ensure_embedding_collection_compatibility() does fail-fast model drift detection. Blue-green adds interrupted-migration detection (check migration_state.json for incomplete migrations).

Why not TaskTracker: The existing TaskTracker (openviking/service/task_tracker.py) is a pure in-memory registry for short-lived background operations (e.g. session commit). Its design explicitly states "v1 is pure in-memory (no persistence). Tasks are lost on restart." Migration runs for hours and must survive restarts -- TaskTracker is fundamentally unsuitable. Migration state is persisted to disk independently.

Design principle: The five-phase flow (dual-write, then bulk re-embed, then query switchover, then disable dual-write, then cleanup) ensures zero data loss. Dual-write is enabled before bulk re-embed starts, so all writes during the migration window enter both collections.

6.1 Migration state persistence

Migration state is persisted separately from #1439's embedding_meta.json -- independent but complementary:

File	Source	Content	Purpose
embedding_meta.json (#1439)	persist_embedding_metadata()	Active embedding identity (provider, model, dimension, mode)	Model drift detection at startup (ensure_embedding_collection_compatibility())
migration_state.json (this proposal)	Migration controller	Blue-green switchover progress (phase, progress, blue/green names)	Incomplete migration detection at startup, supports --resume

# Stored in <workspace>/.meta/migration_state.json
# e.g. ./data/.meta/migration_state.json (workspace defaults to ./data)
{
  "migration_id": "mig_20260417_001",
  "blue_name": "default",
  "green_name": "openai-v3-large",
  "active_name": "default",
  "phase": "building",
  "progress": {
    "total": 47,
    "processed": 12,
    "failed": 0,
    "failed_uris": []
  },
  "started_at": "2026-04-17T10:00:00Z",
  "updated_at": "2026-04-17T10:15:00Z"
}

On server startup, check for incomplete migration and offer resume:

# Server startup detection
WARNING: Incomplete migration detected (mig_20260417_001):
   Phase: building (12/47 resources processed)
   Blue: default, Green: openai-v3-large
   Dual-write: enabled

   Options:
   1. ov reindex --resume        # Continue bulk re-embed from checkpoint
   2. ov reindex --abort         # Disable dual-write, discard green set
   3. ov reindex --all           # Restart from scratch

# If crashed after query switchover:
WARNING: Migration interrupted after query switchover:
   Active: openai-v3-large, Dual-write: was enabled
   Blue set (default) still on disk (for rollback)

   Options:
   1. ov reindex --resume        # Disable dual-write, continue to cleanup
   2. ov reindex --rollback      # Switch queries back to default
   3. ov reindex --abort         # Abort migration entirely

6.2 Interrupted migration recovery

When ov reindex --all is interrupted (Ctrl+C, network disconnect, OOM):

Scenario	Behavior
Graceful stop (SIGINT)	Finish current resource, save state, exit cleanly
Hard crash (OOM, kill -9)	On next startup, detect incomplete state from disk
Network disconnect	Embedder retry logic handles transient failures; persistent failure marks resource as failed and continues
Crash during dual-write	On restart, re-enable dual-write from persisted state, resume bulk re-embed

6.3 Partial failure handling

Individual resource embed failures should not abort the entire migration:

# After migration completes with some failures:
PASS: 45/47 resources migrated successfully
FAIL: 2 resources failed:
   - viking://resources/doc-x (embedder timeout)
   - viking://resources/doc-y (invalid content)

Run `ov reindex --retry-failed` to retry failed resources.

6.4 Disk space pre-check

--dry-run verifies sufficient disk space for the green collection:

ov reindex --all --dry-run
# Output:
#   Resources to reindex: 47
#   Estimated green collection size: ~2.3 GB
#   Available disk space: 15.7 GB
#   Estimated time: ~12 min

6.4.1 Migration state writability check

--dry-run also verifies that migration_state.json can be written and persisted:

ov reindex --all --dry-run
# Output:
#   Resources to reindex: 47
#   Estimated green collection size: ~2.3 GB
#   Available disk space: 15.7 GB
#   Estimated time: ~12 min
#
#   Migration state:
#   PASS: State path writable: <workspace>/.meta/migration_state.json
#   PASS: Test write successful -- state will survive restarts

# WARNING: container with read-only mount
ov reindex --all --dry-run
# Output:
#   ...
#   Migration state:
#   FAIL: State path NOT writable: <workspace>/.meta/migration_state.json
#      Reason: Read-only filesystem
#
#   WARNING: If workspace is on a read-only mount,
#      migration state will be lost on container restart.
#      A full reindex will be required if migration is interrupted.
#
#   To fix: ensure workspace directory is on a persistent writable volume

6.5 Rollback TTL

Blue collection retention after switchover is configurable:

// ov.conf
{
  "embedding": {
    "migration": {
      "rollback_ttl_hours": 72,    // Delete blue set after 72h if not rolled back
      "auto_confirm": false         // If true, auto-cleanup after TTL without admin confirmation
    }
  }
}

Alternatives Considered

1. Status quo (manual, in-place overwrite) -- works but tedious, and users see search quality degradation during the migration window. Unacceptable. Additionally, changing model or dimension requires manually dropping and recreating the entire vectordb -- a destructive operation.
2. Automatic reindex on config change -- too risky. A typo could trigger an expensive reindex. Better to keep it explicit via ov reindex --all.
3. Block search during migration -- too aggressive. Blue-green achieves zero downtime.
4. Expose migration state in API responses -- leaks internal operational details to users. Blue-green keeps migration invisible to users, which is the better approach.

Expected Outcome

Who	Before	After
Admin	Manual config edit, restart without validation, individual reindex, no progress, no rollback	ov config validate --live, then ov reindex --all --dry-run, then ov reindex --all with progress and --rollback
Admin	Changing model/dimension requires dropping and recreating the entire vectordb	Blue-green: changing model and changing dimension are the same operation
Admin	Migration progress lost on interruption, no state after container restart	Migration state persisted to <workspace>/.meta/, --resume continues from checkpoint
User	Search quality degrades during migration (half-old-half-new vectors)	Zero user-visible impact -- search always hits a complete, consistent vector set

Related Issues

• feat: detect embedding model drift and add rebuild tool #1439 -- [Open PR] feat: detect embedding model drift and add rebuild tool. This proposal's prerequisite: provides embedding_meta.json persistence, compatibility_identity(), VectorRebuildService, openviking-rebuild-vectors CLI.
• Feature: auto-detect embedding model change and trigger vector rebuild #1066 -- Feature: auto-detect embedding model change and trigger vector rebuild (closed by feat: detect embedding model drift and add rebuild tool #1439)
• [Bug]: 0.3.5使用qwen的embedding模型出错 #1442 -- [Closed, fixed in v0.3.6] [Bug]: 0.3.5 using qwen embedding model error
• [Feature] reindex should support recursive directory traversal #1073 -- [Feature] reindex should support recursive directory traversal (overlaps with --all proposal)
• Feature request: make text file vectorization strategy configurable to avoid embedding oversize failures #857 -- Feature request: make text file vectorization strategy configurable to avoid embedding oversize failures

Feature Area

Configuration / CLI / Server Runtime

Contribution

• I am willing to contribute to implementing this feature

[A0nameless0man]

Rethinking the metadata storage: decoupling from #1439

After reconsidering, I think the hard dependency on #1439 is a mistake. #1439 is an unreviewed PR from an automated bot account that has opened 16 PRs in a single day across unrelated modules. It may never merge, or it may merge with substantial changes to its API surface. Either way, anchoring this proposal to its specific interfaces (embedding_meta.json format, compatibility_identity(), VectorRebuildService) creates an unnecessary coupling that blocks forward progress.

The actual state of metadata today

I audited the current vectordb metadata structures. Here's what actually exists:

C++ layer (src/index/detail/meta/vector_index_meta.h):

class VectorIndexMeta {
  std::string distance_type = "ip";
  std::string index_type;
  std::string quantization_type = "float";
  uint64_t element_count = 0;
  uint64_t max_element_count = 0;
  uint64_t dimension = 0;           // ← dimension IS stored
  // no embedding_model, no provider, no version
};

C++ layer (src/index/detail/meta/manager_meta.h):

class ManagerMeta {
  std::string collection_name;
  std::string index_name;
  uint64_t update_timestamp = 0;
  std::string vector_index_type;
  // no embedding identity at all
};

Python layer (openviking/storage/vectordb/meta/index_meta.py):
Serializes dimension from collection metadata. No model identity.

Key takeaway: The only embedding-related metadata currently persisted is dimension. There is no embedding_model, no provider, no way to tell which model produced the vectors sitting in the collection. This means we need to add embedding identity storage regardless of whether #1439 merges — it's a gap that exists independently of any single PR.

Why not extend the existing metadata structures

Before proposing a new storage scheme, I considered whether embedding identity could be added to the existing metadata layer. After auditing the code, the answer is no — it's the wrong layer.

The vectordb metadata is split across two layers:

C++ engine layer (src/index/detail/meta/):

• ManagerMeta → persisted as manager_meta.json by IndexManagerImpl::dump(), read back by load_from_path(). Contains collection_name, index_name, update_timestamp, and the VectorIndexMeta sub-structure.
• VectorIndexMeta → physical index properties: index_type, dimension, distance_type, quantization_type, element_count. Written via rapidjson, read via init_from_json().
• These are engine internals. Modifying them means changing the C++ source, updating the abi3 bindings, and recompiling the native extension. That's a high-risk change for something the engine doesn't need to know about — the engine doesn't care which model produced the vectors, only that they're floats with a certain dimension.

Python application layer (openviking/storage/vectordb/meta/):

• CollectionMeta → persisted via PersistentDict (backed by FileStore). Describes the collection schema: Fields, Dimension, PrimaryKey, VectorKey.
• IndexMeta → persisted via PersistentDict. Describes the index configuration: VectorIndex { IndexType, Dimension, Distance, Quant }, ScalarIndex.
• Both use IDict → PersistentDict for atomic JSON persistence. The PersistentDict.override() path does a full JSON serialize + FileStore.put() on every update.

Embedding identity (provider, model name) doesn't belong in either:

• It's not a physical index property (wrong for C++ layer).
• It's not a collection schema property — it's a runtime annotation about how the data was produced. Collections don't have an intrinsic "model" — the same collection could theoretically be re-indexed with a different model entirely. Putting it in CollectionMeta or IndexMeta conflates schema with provenance.

Proposed change: sidecar identity file

The right approach is a lightweight sidecar file that lives next to the collection data but is independent of the schema/engine metadata. Python-layer only, no C++ changes, no schema modifications.

<workspace>/                          # defaults to ./data
├── vectordb/
│   ├── context/                      # collection directory (blue, active)
│   │   ├── manager_meta.json         # C++ engine metadata (existing, untouched)
│   │   ├── ...
│   │   └── embedding_identity.json   # sidecar: who produced these vectors
│   └── context_green/                # green collection (migration target)
│       ├── manager_meta.json
│       ├── ...
│       └── embedding_identity.json   # different identity from blue
├── migration_state.json              # migration progress (only while active)
└── ...

Per-collection, not global. In a blue-green setup, the "blue" and "green" collections have different embedding identities simultaneously. A global file can only describe one.

Why identity is not a single provider/model pair

The current EmbeddingConfig supports three embedder slots — dense, sparse, and hybrid — each an independent EmbeddingModelConfig with its own provider, model, and dimension. The instantiation priority (from get_embedder() in embedding_config.py) is:

1. hybrid set → single model returning both dense + sparse vectors
2. dense + sparse both set → CompositeHybridEmbedder wrapping two separate embedders
3. dense only → plain dense embedder

This means the identity of a collection's vectors isn't one model — it's whichever subset of these slots was active when the vectors were written. The sidecar file needs to capture all of them:

// <workspace>/vectordb/<collection>/embedding_identity.json
{
  // Which embedder mode produced these vectors
  // "dense" | "hybrid" | "composite" (dense+sparse)
  "mode": "dense",

  // Identity for each active slot. Not all slots are present —
  // only the ones that were active when vectors were written.
  "dense": {
    "provider": "volcengine",
    "model": "doubao-embedding-vision-251215",
    "dimension": 1024,
    "input": "multimodal"
  },
  // "sparse": { ... },   — present only for composite mode
  // "hybrid": { ... },   — present only for single hybrid model

  // When this identity was written (for drift detection)
  "written_at": "2026-04-17T10:00:00Z"
}

The mode field records which get_embedder() branch was active, so we know how to interpret the rest. This handles all current and near-future scenarios:

Config	mode	Identity slots present
dense only	"dense"	dense
dense + sparse	"composite"	dense, sparse
hybrid	"hybrid"	hybrid

Uses the same persistence mechanism as PersistentDict — a plain JSON file written atomically via FileStore.put(). No new infrastructure needed.

Why no config_hash field? I considered a hash for quick equality checks, but it adds complexity without much benefit. The identity file is tiny (<300 bytes), and comparing a few string fields is already O(1). If someone really wants it later, it's an additive change.

Relationship to existing metadata

File	Owner	Lifecycle	Contains
manager_meta.json	C++ engine (IndexManagerImpl::dump())	Written on every dump, read on load	Index physical structure: type, dimension, quantization, element count
collection_meta.json	Python CollectionMeta via PersistentDict	Written on create/update	Collection schema: fields, primary key, vector key
index_meta.json	Python IndexMeta via PersistentDict	Written on create/update	Index config: distance, scalar fields
embedding_identity.json	Python migration layer (this proposal)	Written on first index build or after migration	Embedding provenance: mode, per-slot provider/model/dimension

No overlap. The engine's dimension and the identity file's dimension are intentionally duplicated — the engine value is the physical truth (what the index was built with), the identity value is the semantic truth (what config produced it). They should always agree; if they don't, that's corruption.

Migration state file

Lives at the workspace root (no .meta/ subdirectory needed):

<workspace>/migration_state.json

The file only exists while a migration is active; it's deleted after cleanup. Crucially, it stores full identity snapshots for both blue and green — not just collection names. This makes it self-contained: even if ov.conf is read-only (e.g., container mount) or gets overwritten by orchestration tools, the migration can continue from migration_state.json alone.

// <workspace>/migration_state.json
{
  "phase": "building",  // dual_write_enabled | building | switched | dual_write_disabled | cleanup_done | failed
  "blue_name": "context",
  "green_name": "context_green",

  // Identity snapshots — the runtime truth source, independent of ov.conf
  "blue_identity": {
    "mode": "dense",
    "dense": { "provider": "volcengine", "model": "doubao-embedding-vision-251215", "dimension": 1024 }
  },
  "green_identity": {
    "mode": "dense",
    "dense": { "provider": "dashscope", "model": "text-embedding-v3", "dimension": 1024 }
  },

  "started_at": "2026-04-17T10:00:00Z",
  "progress": { "total": 5000, "embedded": 2300 }
}

Note: ov.conf can declare the migration target via embedding.migration.<name> (as discussed in the issue), but the migration controller reads its truth from migration_state.json. The config is a trigger, not the authority.

Startup consistency check

Four sources of truth exist during a migration:

① blue embedding_identity.json    — what's actually in the blue collection
② green embedding_identity.json   — what's actually in the green collection
③ migration_state.json            — what the migration controller recorded
④ ov.conf embedding section       — what the admin configured

Key design decision: migration_state.json is the runtime authority. ov.conf may be read-only (container mount), but migration_state.json is always in the writable workspace. A container restart can resume the migration even if the config mount was reset.

No migration in progress — check config vs stored identity (the simpler check_embedding_drift()):

Check	Severity	Reason
Mode changed (dense → hybrid, etc.)	Hard error	Structural change in how vectors are stored
Dimension mismatch in any active slot	Hard error	Vectors are fundamentally incompatible
Model/provider change in any active slot	Warning	Vectors might still be usable (e.g., same-provider upgrade)
No identity file exists yet	Write it	First run

Migration active — cross-validate all four sources:

Check	Severity	Reason
① ≠ ③ (blue identity doesn't match migration_state)	Hard error	Blue collection was externally modified
② ≠ ③ (green identity doesn't match migration_state)	Hard error	Green collection was externally modified
④ ≠ ③ (ov.conf doesn't match blue identity in migration_state)	Warning only	ov.conf may be a read-only mount; migration_state is the authority

After validation passes, log migration state (phase, blue/green names, progress) and resume.

What this removes from the proposal

Originally from #1439	Replacement
embedding_meta.json (global, single model)	embedding_identity.json (per-collection sidecar, multi-slot)
compatibility_identity()	identity_from_config() — extract mode + per-slot provider/model/dimension from EmbeddingConfig
ensure_embedding_collection_compatibility()	Two-tier startup check: drift detection (no migration) + four-source consistency validation (migration active)
VectorRebuildService	Not needed — blue-green migration replaces the in-place rebuild model entirely
openviking-rebuild-vectors CLI	Replaced by ov reindex --all (Section 1)
EmbeddingCompatibilityError	Keep as simple exception classes (EmbeddingModeMismatch, EmbeddingDimensionMismatch, MigrationCorruptionError)
embedding_compat.py (new module)	No new module needed — identity read/write is ~40 lines, lives in the migration controller
vector_rebuild.py (new service)	No new service — blue-green controller handles rebuild with different semantics

The VectorRebuildService deserves a note. #1439 designs it as a service that does in-place delete + rebuild. Blue-green migration fundamentally replaces that model — instead of deleting and rebuilding in-place, we write to a new collection and atomically switch. The discover_accounts() and _discover_directories() traversal logic from #1439 is useful, but it's ~50 lines of directory traversal that we can write ourselves rather than depending on an unreviewed +820-line PR for it.

Why this is better

1. No blocking dependency — this proposal is self-contained and can proceed independently.
2. Per-collection identity — naturally supports blue-green (two collections with different identities simultaneously). A global embedding_meta.json only describes one identity, which is the wrong abstraction for a migration scenario.
3. Multi-slot identity — correctly handles the actual EmbeddingConfig structure (dense/sparse/hybrid), not just a single model. The original proposal and feat: detect embedding model drift and add rebuild tool #1439 both treat embedding identity as one provider+model pair, which breaks for composite and hybrid modes.
4. Simpler scope — the startup drift check is ~30 lines. It doesn't justify a separate +820-line PR prerequisite.
5. Container-safe — migration_state.json is the runtime authority, not ov.conf. Works with read-only config mounts.
6. feat: detect embedding model drift and add rebuild tool #1439 can still merge independently — if it does, the two approaches can be reconciled later (converge on a shared identity format). But this proposal shouldn't be blocked on that.

What changes in the proposal

• Section 4 (Runtime health gate): Replace feat: detect embedding model drift and add rebuild tool #1439 reference with the self-contained check_embedding_drift() + verify_startup_consistency() above. Update to check mode + per-slot identity, not just a single provider/model pair.
• Section 6 (Migration resilience): Remove feat: detect embedding model drift and add rebuild tool #1439 dependency declaration. The migration state file now stores full identity snapshots (not just collection names), making it independent of ov.conf.
• Proposed Solution header: Remove the dependency block. This proposal no longer requires any prerequisite PR.
• Related Issues: feat: detect embedding model drift and add rebuild tool #1439 becomes a reference (not a prerequisite), along with Feature: auto-detect embedding model change and trigger vector rebuild #1066 which it aims to close.

Thoughts?