feat(index): optional Milvus derived-index backend behind a typed port

.github/workflows/ci.yml

@@ -126,6 +126,59 @@ jobs:
       - name: Integration tests
         run: make integration
 
+  integration-milvus:
+    name: integration tests (Milvus 2.6)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v8.2.0
+        with:
+          enable-cache: true
+          cache-dependency-glob: uv.lock
+
+      - name: Set up Python
+        run: uv python install 3.12
+
+      - name: Install dependencies (frozen)
+        run: make install-deps
+
+      - name: Start Milvus 2.6.22
+        run: |
+          curl --fail --location --retry 3 \
+            --output /tmp/milvus-compose.yml \
+            https://github.com/milvus-io/milvus/releases/download/v2.6.22/milvus-standalone-docker-compose.yml
+          docker compose -f /tmp/milvus-compose.yml up -d
+          for attempt in {1..60}; do
+            if curl --fail --silent http://127.0.0.1:9091/healthz >/dev/null; then
+              exit 0
+            fi
+            sleep 2
+          done
+          docker compose -f /tmp/milvus-compose.yml logs
+          exit 1
+
+      - name: Milvus repository contract
+        env:
+          EVEROS_TEST_MILVUS_URI: http://127.0.0.1:19530
+          EVEROS_TEST_MILVUS_FULL_STARTUP: "1"
+        run: uv run --frozen pytest tests/integration/test_milvus_remote.py -v
+
+      # The contract tests above exercise the repository port directly. This
+      # drives the same request path the application does -- /add, /flush,
+      # cascade, every search method, /get -- against the remote backend, so a
+      # break anywhere between the API and Milvus surfaces here rather than in
+      # production. Deselecting the LanceDB half keeps the job to its subject.
+      - name: Milvus end-to-end (tiered API suites)
+        env:
+          EVEROS_TEST_MILVUS_URI: http://127.0.0.1:19530
+        run: uv run --frozen pytest tests/integration/test_tiers -v -k milvus
+
+      - name: Stop Milvus
+        if: always()
+        run: docker compose -f /tmp/milvus-compose.yml down -v
+
   package:
     name: package build
     runs-on: ubuntu-latest

config.example.toml

@@ -56,3 +56,15 @@ max_concurrent  = 5
 #
 # [lancedb]
 # read_consistency_seconds = 5.0
+#
+# [index]
+# backend = "lancedb"  # or "milvus"
+#
+# # Required only when index.backend = "milvus". Install with:
+# #   pip install "everos[milvus]"
+# [milvus]
+# uri = "http://127.0.0.1:19530"  # Milvus Server or Zilliz Cloud endpoint
+# token = ""                       # e.g. "root:Milvus" or cloud token
+# db_name = ""
+# consistency_level = "Session"
+# collection_prefix = "everos"

docs/api.md

@@ -93,8 +93,9 @@ bare FastAPI `detail`); see [Errors](#errors).
 
 `/add` and `/flush` write the markdown file (the source of truth)
 **synchronously** — when the call returns with `status: "extracted"`,
-the new entry exists on disk. The LanceDB vector / BM25 / scalar index
-is rebuilt by the in-process **cascade coroutine asynchronously**.
+the new entry exists on disk. The configured vector / BM25 / scalar
+index backend is rebuilt by the in-process **cascade coroutine
+asynchronously**.
 
 That means `/search` and `/get` may not see a record immediately after
 the `/flush` that produced it. Typical sync latency is sub-second, but
@@ -371,7 +372,7 @@ A recursive boolean tree of predicates. Used by `/search.filters` and
 `/get.filters`. The Pydantic envelope only checks the recursive
 combinator shape; field-level validity (which scalar fields are
 filterable, which operators apply, value coercion) runs when the
-node is compiled to a LanceDB `where` clause server-side. Compile
+node is compiled to a backend-specific filter clause server-side. Compile
 errors surface as `422` with the offending field / operator in
 `error.message`.
 
@@ -464,12 +465,12 @@ Examples:
 |---|---|
 | `"keyword"` | BM25 only — pure lexical match, no embedding cost |
 | `"vector"` | Dense vector ANN only — semantic recall, no lexical |
-| `"hybrid"` *(default)* | Reciprocal-rank fuse of BM25 + vector + optional scalar filter in a single LanceDB query |
+| `"hybrid"` *(default)* | Reciprocal-rank fuse of BM25 + vector + optional scalar filter against the configured derived index backend |
 | `"agentic"` | Iterative cluster-path retrieval driven by a cross-encoder rerank loop; higher quality at higher latency / cost |
 
 `"hybrid"` is the default because it balances recall and precision
-with one LanceDB roundtrip. `"agentic"` calls the LLM in a loop and
-should be reserved for offline or background workflows.
+without requiring the agentic loop. `"agentic"` calls the LLM in a loop
+and should be reserved for offline or background workflows.
 
 ### GetMemoryType
 
@@ -613,7 +614,7 @@ scope.
   this `(session_id, app_id, project_id)`, or it was already flushed).
 
 `/flush` is synchronous with respect to markdown persistence: by the
-time the response returns, the new entry is on disk. LanceDB index
+time the response returns, the new entry is on disk. Derived index
 sync is still asynchronous — see
 [Eventual consistency](#eventual-consistency).
 

docs/architecture.md

@@ -17,7 +17,7 @@
 │    + reflection + strategies + get + events            │
 ├──────────────────────────────────────────────────────┤
 │  infra/persistence  (Storage adapters; infra/ may host other adapter types)    │
-│    markdown + sqlite + lancedb                        │
+│    markdown + sqlite + derived index                  │
 └──────────────────────────────────────────────────────┘
 
 Cross-cutting (used by all layers, depends on none):
@@ -68,8 +68,8 @@ layers = [
 └────────────────────────────────────────────────────────────────┘
 
    ┌──────────────┐   ┌──────────────┐   ┌─────────────────┐
-   │   Markdown   │   │   SQLite     │   │    LanceDB      │
-   │  (truth)     │   │  (state)     │   │  (index)        │
+   │   Markdown   │   │   SQLite     │   │ Derived index   │
+   │  (truth)     │   │  (state)     │   │ LanceDB/Milvus  │
    ├──────────────┤   ├──────────────┤   ├─────────────────┤
    │ entries +    │   │ change queue │   │ vector ANN      │
    │ frontmatter  │   │ + state/LSN  │   │ BM25 (Tantivy)  │
@@ -78,7 +78,7 @@ layers = [
    └──────────────┘   └──────────────┘   └─────────────────┘
           │                  │                    │
           ▼                  ▼                    ▼
-    memory-root/         .index/sqlite/      .index/lancedb/
+    memory-root/         .index/sqlite/      .index/<backend>/
    (truth source)       (system data)       (rebuildable)
 ```
 
@@ -101,10 +101,13 @@ External message
    │        │
    ▼        ▼
 4a. SQLite   4b. memory.cascade  (async daemon)
-    audit        watches md → diff entries → LanceDB sync
+    audit        watches md → diff entries → index sync
 ```
 
-**Key guarantee**: md write is strongly consistent (fsync). LanceDB is eventually consistent. LanceDB unavailability does not block response — changes buffer in the SQLite `md_change_state` queue, replayed on recovery.
+**Key guarantee**: md write is strongly consistent (fsync). The derived
+index is eventually consistent. Index backend unavailability does not block
+response — changes buffer in the SQLite `md_change_state` queue, replayed on
+recovery.
 
 ## Read path
 
@@ -115,8 +118,8 @@ User query
 1. service.search
    │
    ▼
-2. memory.search (hybrid)     single LanceDB query =
-                                BM25 + vector ANN + scalar filter
+2. memory.search (hybrid)     BM25 + vector ANN + scalar filter
+                              through the configured index backend
    │
    ▼
 3. (optional) read md         original markdown for context
@@ -139,12 +142,13 @@ extract/
 
 ### `memory/cascade/`
 
-Daemon that watches markdown changes and syncs to LanceDB:
+Daemon that watches markdown changes and syncs to the configured derived
+index backend:
 
 - inotify / FSEvents file watcher (cross-platform via `watchdog`)
 - 500ms debounce
 - Entry-level diff (added / changed / removed)
-- LanceDB single-transaction update (text + vector columns atomic)
+- Per-entry index upsert / delete (text + vector columns update together)
 - LSN-based crash recovery via the SQLite `md_change_state` queue
 - Handlers for all eight business kinds: episode, atomic_fact, foresight,
   user_profile, agent_case, agent_skill, knowledge_document, knowledge_topic
@@ -225,7 +229,7 @@ holding **only memory extraction algorithms**:
 everalgo is:
 
 - **Stateless** — pure functions, no class hierarchy
-- **No I/O** — does not touch md files / LanceDB / SQLite
+- **No I/O** — does not touch md files, derived indexes, or SQLite
 - **No prompts inline** — extractors that accept a prompt-override parameter use the project-supplied value; others use their algo-bundled defaults
 
 This boundary lets everalgo be reused across product forms (this open-source build, EverOS Cloud, OpenClaw plugins, etc.).

docs/cascade_runbook.md

@@ -1,9 +1,13 @@
 # Cascade Runbook
 
-The cascade daemon keeps LanceDB in sync with the markdown files under
-the memory root. Service / entry points only ever write markdown; the
-daemon is the **sole** writer of the LanceDB index. This runbook covers
-the recurring operational questions.
+The cascade daemon keeps the configured derived index in sync with the
+markdown files under the memory root. Service / entry points only ever write
+markdown; the daemon is the **sole** writer of the derived index. This runbook
+covers the recurring operational questions.
+
+Sections that mention LanceDB-specific schemas, index cache, file descriptors,
+or `lance error` messages apply to the default LanceDB backend. Milvus uses the
+same cascade queue with backend-specific collection management.
 
 ## What runs where
 
@@ -13,7 +17,7 @@ providers in order:
 1. **Metrics** — Prometheus collector.
 2. **LLM** — LLM client initialisation.
 3. **SQLite** — system DB + schema (`SQLModel.metadata.create_all`).
-4. **LanceDB** — async connection + schema verification + FTS indexes.
+4. **Derived index** — async connection + schema verification + search indexes.
 5. **Cascade** — watcher + scanner + worker, all in-process tasks.
 6. **OME** — offline memory engine.
 
@@ -23,7 +27,7 @@ The cascade subsystem itself is three independent loops:
 |---|---|---|
 | Watcher | `watchdog` filesystem events (sync thread) | `md_change_state.upsert` per registered kind |
 | Scanner | Periodic walk (`scan_interval_seconds`, default 30 s) | Same — catches changes the watcher missed |
-| Worker | `claim_pending_batch` polling (default 1 s when idle) | Handler dispatch → LanceDB upsert / delete |
+| Worker | `claim_pending_batch` polling (default 1 s when idle) | Handler dispatch → index upsert / delete |
 
 Every loop talks to the same `md_change_state` sqlite table. The
 worker's claim mode (`pending → processing → done/failed`) keeps
@@ -123,7 +127,7 @@ parallel with a live `everos server`.
 
 ## Rebuild the index: `everos cascade rebuild`
 
-The safe recovery from a drifted or corrupt LanceDB index. It rebuilds
+The safe recovery from a drifted or corrupt derived index. It rebuilds
 the whole index from markdown (the source of truth) in one shot:
 
 ```bash
@@ -132,15 +136,16 @@ everos cascade rebuild --yes    # non-interactive
 ```
 
 > **Stop the `everos server` first.** Unlike `cascade sync`, rebuild
-> **drops and recreates** the LanceDB tables. A running daemon holds
+> **drops and recreates** the active backend's tables or collections. A running
+> daemon holds
 > cached table handles that would keep pointing at (and writing to) the
 > dropped dataset, corrupting the rebuild. This is the one cascade
 > command that is **not** safe to run alongside a live server.
 
 What it does, in order:
 
-1. **Drops** every business LanceDB table (`drop_business_tables`) and
-   evicts them from the connection cache.
+1. **Drops** every business table or collection (`drop_business_tables`) and
+   evicts it from the process cache.
 2. **Recreates** them empty from the current schema + FTS indexes
    (`ensure_business_indexes`).
 3. **Clears** the cascade queue (`md_change_state.reset_all`) so every
@@ -160,6 +165,10 @@ Why not a bare `rm`:
 | `rm -rf .index` | ✅ | ❌ deletes un-extracted messages |
 | `everos cascade rebuild` | ✅ | ✅ |
 
+For a remote Milvus backend, rebuild acts only on collections whose names use
+the configured `collection_prefix`. Use a unique prefix or dedicated database
+before running it on a shared Milvus Server or Zilliz Cloud deployment.
+
 ## Recovery paths
 
 ### LanceDB schema drift on startup

docs/configuration.md

@@ -99,6 +99,29 @@ everos init --root /data/everos
 | `read_consistency_seconds` | float \| null | `null` | Read consistency interval. `null` = no check, `0` = strict, `>0` = eventual. |
 | `index_cache_size_bytes` | int | `16777216` | Upper bound on LanceDB index cache (16 MB default). |
 
+### `[index]`
+
+The derived vector/BM25 index is rebuildable; Markdown remains the source of
+truth regardless of backend.
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `backend` | string | `"lancedb"` | Index implementation: `lancedb` or `milvus`. |
+
+### `[milvus]`
+
+Used only when `index.backend = "milvus"`. Install the optional client with
+`pip install "everos[milvus]"`. EverOS supports a remote Milvus Server or
+Zilliz Cloud endpoint; a Milvus Lite filesystem path is rejected.
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `uri` | string | `""` | Milvus Server or Zilliz Cloud URI. |
+| `token` | string | `""` | Authentication token, when required. |
+| `db_name` | string | `""` | Optional Milvus database name. |
+| `consistency_level` | string | `"Session"` | Milvus consistency level used by collections. |
+| `collection_prefix` | string | `"everos"` | Prefix for the seven derived-index collections. |
+
 ### `[llm]`
 
 | Field | Type | Default | Required | Description |

docs/knowledge.md

@@ -48,8 +48,8 @@ Each level corresponds to a different granularity of API:
 ## Storage layout
 
 Every document is a self-contained directory. Markdown files are the
-single source of truth; SQLite and LanceDB are derived indexes built
-automatically by the cascade daemon.
+single source of truth; SQLite and the configured vector/BM25 index are
+derived indexes built automatically by the cascade daemon.
 
 ```
 ~/.everos/<app>/<project>/knowledge/
@@ -73,16 +73,16 @@ automatically by the cascade daemon.
 ### Storage roles
 
 ```
-Markdown (source of truth)  +  SQLite (structured state)  +  LanceDB (vector + BM25 index)
+Markdown (source of truth)  +  SQLite (structured state)  +  derived vector/BM25 index
 ```
 
 | Store | What it holds | Role |
 |-------|---------------|------|
 | Markdown | Document metadata, summaries, topic content, original files | Single source of truth; human-readable and editable |
 | SQLite | Document rows, topic rows (with content), change queue | Structured queries, paginated lists, count aggregation |
-| LanceDB | Topic vectors, BM25 tokens, scalar fields | Search index (fully rebuildable from Markdown) |
+| Derived index | Topic vectors, BM25 tokens, scalar fields | Search index (fully rebuildable from Markdown) |
 
-Even if SQLite and LanceDB data is corrupted, as long as the Markdown
+Even if SQLite and derived index data is corrupted, as long as the Markdown
 files are intact, the indexes can be fully rebuilt via the cascade daemon.
 
 ### Markdown format
@@ -500,7 +500,7 @@ query ─→ embed ─→ keyword (BM25) ─┐
 ```
 
 1. **Embed** — the query is embedded using the configured embedding provider
-2. **Recall** — dual-channel retrieval from LanceDB:
+2. **Recall** — dual-channel retrieval from the configured derived index:
    - BM25 channel: keyword matching on `summary_tokens` + `content_tokens`
    - ANN channel: nearest-neighbor search on the `vector` column
    - In `hybrid` mode, both channels run in parallel
@@ -531,15 +531,15 @@ export EVEROS_KNOWLEDGE__SEARCH__RERANK_N=100
 ## Cascade sync
 
 The cascade daemon watches the knowledge Markdown directory for file
-changes and keeps SQLite + LanceDB in sync.
+changes and keeps SQLite + the configured derived index in sync.
 
 ```
 md file written
   → FSEvents / watchdog detects change
   → worker picks up from queue (≤1s poll interval)
   → handler dispatched by file type:
       index.md    → KnowledgeDocumentHandler → SQLite upsert (metadata)
-      N_topic.md  → KnowledgeTopicHandler    → tokenize + embed + SQLite + LanceDB upsert
+      N_topic.md  → KnowledgeTopicHandler    → tokenize + embed + SQLite + index upsert
 ```
 
 The topic handler uses a SHA-256 content digest to skip unchanged files —
@@ -597,7 +597,7 @@ curl .../documents?app_id=tenant_b&project_id=proj_1
 # → { "documents": [], "total": 0 }
 ```
 
-Storage paths, SQLite rows, and LanceDB indexes are all scoped by
+Storage paths, SQLite rows, and derived index rows are all scoped by
 `app_id` + `project_id`.
 
 ## End-to-end walkthrough

docs/multimodal.md

@@ -41,7 +41,7 @@ POST /api/v2/memory/add
   boundary detector → extraction LLM → memory cell (MemCell)
         │
         ▼
-  markdown (truth)  +  SQLite (state)  +  LanceDB (vector + BM25)
+  markdown (truth)  +  SQLite (state)  +  derived vector/BM25 index
         │
         ▼
   retrievable via /search and /get like any text memory