feat(index): optional Milvus derived-index backend behind a typed port

[gloryfromca]

Summary

Adds an optional Milvus backend for the rebuildable derived index, behind a
typed port that LanceDB also implements.

This continues @zc277584121's work in #332 rather than replacing it. That
branch is included here commit-for-commit — it introduced the Predicate AST,
the portable schema derivation, the Milvus repository, and migrated the filter
DSL and the recallers off rendered LanceDB SQL. The four commits on top
restructure the layer those pieces sit in and fix defects found reviewing it.

What the follow-up commits change

refactor(index): split the derived-index port into contracts, routing, and a Lance adapter — three gaps motivated it. The repository surface did not
include maintenance operations, so the cascade worker could not schedule
optimize / prune / rebuild without knowing the active backend. Backfill still
reached LanceDB directly for its null-vector scan and its cluster-phase
full-table read. And there was no typed port a second adapter could be checked
against. Adds index/protocols.py (the ports), index/router.py (stable
repository identities, backend resolved per call), and index/lancedb.py (the
LanceDB adapter, including the predicate renderer).

refactor(index): stop the port leaking backend limits and syntax — a
Milvus/Zilliz topK ceiling had been hard-coded into memory/search, which also
shrank LanceDB's fact-expansion pool to 1024; per-column BM25 lost its
asyncio.gather. Both restored. Empty all_of() / any_of() now raise
instead of rendering to "" (an empty filter matches every row, and for OR it
inverts the meaning). The import-privacy contract is extended to index.**
and milvus.**, and the filter-DSL tests now assert against the predicate tree
instead of re-rendering it through the LanceDB adapter.

fix(milvus): correct the datetime round-trip, update race, and scan window —
to_timestamp_ms writes milliseconds unconditionally, but the read path used
from_timestamp, which treats anything under 1e12 as seconds: every instant
before 2001-09-09 came back in the year 30000 or raised. update() did
read-modify-write with only the upsert under the write lock, so concurrent
writers to one row erased each other's columns. list_by_owner_after_ts asked
client.query for 20,000 rows, which cannot cross Milvus' 16,384-row window.

fix(milvus): verify the physical collection schema, not just field names —
verify_collection() compared field names only, so a collection whose
columns shared our names but carried the wrong datatype, primary key,
nullability or vector dimension passed startup and failed opaquely on the first
write. Creation and verification now share one descriptor; datatype, primary,
nullable, dimension and element type are strict, while VARCHAR/array limits are
advisory (the server may normalize them, and a wrong length already fails
loudly on write).

Area

• Architecture method
• Benchmark
• Use case
• Documentation
• Developer experience
• CI, build, or release

Verification

make lint          all checks passed; 3 import contracts kept
make test          2118 passed
make integration   183 passed, 4 skipped, 7 deselected

The 4 skips are tests/integration/test_milvus_remote.py, which needs
EVEROS_TEST_MILVUS_URI. They have NOT been run locally — no Milvus was
available. The integration-milvus job in this PR is their first execution.

Checklist

• I kept the change scoped to the relevant area.
• I am opening this from a separate branch, not pushing directly to main.
• I updated docs, examples, or setup notes when behavior changed.
• I added or updated tests when the change affects behavior.
• I did not commit secrets, .env files, dependency folders, or generated output.
• Active relative links in Markdown files resolve.

Notes for Reviewers

Please review #332's commits and the follow-ups separately. The physical
Milvus adapter (infra/persistence/milvus/**) is @zc277584121's work and is
best reviewed by someone who knows Milvus; the port, routing and Lance adapter
are the EverOS side.

Four remote Milvus tests have never run. The integration-milvus job is
the first real execution of test_milvus_remote.py for this branch, and three
of its cases are new here: update() on a row that carries vectors (previous
coverage used user_profile, which has no vector column), a pre-2001 timestamp
round-trip, and re-verifying every collection after clearing the process
readiness cache. That last one is the only check that can confirm the physical
schema descriptor matches what Milvus actually reports — every offline test
builds its fake reply from the same descriptor the verifier reads, so those can
only prove creation and verification agree with each other.

Deliberately not included:

• Index and metric verification (dense COSINE, BM25 output fields). It needs
  a second descriptor shape (describe_index) that could not be confirmed
  without a live server, and an unverifiable check risks exactly the false
  positive the rest of this change is careful to avoid.
• Moving the record models out of infra/persistence/lancedb. The Milvus
  adapter still imports them from there, so the schema is not yet
  backend-neutral in the strict sense. Doing it here would change
  to_arrow_schema() output and force a rebuild for existing deployments.
• Milvus soak testing. The 18,000-second soak in this work covered LanceDB
  only; it is not a Milvus long-run proof.

Known limits carried forward: update() on Milvus now holds the write lock
across its read, which serializes updates against other writes on that
collection; scan() returns a list, so a very large backfill materializes in
memory; the pagination contract still tops out at a 20,000-candidate window.

[gloryfromca]

Superseded by #332 — the same nine commits now live on that branch, pushed as a fast-forward so @zc277584121's five commits keep their SHAs and authorship.

Consolidating there rather than here because collaboration needs push access: on this PR that would have meant repo-wide write for an external contributor, whereas on their own fork they already have it. #332 stays their PR and we can both push to it.