Skip to content

fix(routing): keep embedder fingerprint stable across probed reasoning - #12566

Open
walcz-de wants to merge 1 commit into
mudler:masterfrom
walcz-de:fix/embedder-fingerprint-probed-reasoning
Open

walcz-de wants to merge 1 commit into
mudler:masterfrom
walcz-de:fix/embedder-fingerprint-probed-reasoning

Conversation

@walcz-de

@walcz-de walcz-de commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Description

No upstream issue exists for this; the bug is described here.

EmbedderFingerprint hashes the whole model config of the embedding model behind a knn router. A chat completion sent to that embedding model (with use_tokenizer_template set) runs the thinking probe, and persistProbedReasoning writes the probed reasoning slots (DisableReasoning, DisableReasoningTagPrefill) back into the shared config. That single request moved the fingerprint, and from then on every knn router built on the embedder answered with ErrLiveEmbeddingMismatch (HTTP 503) until LocalAI was restarted.

Reasoning settings never change the embedding space, so this change leaves ReasoningConfig out of the hashed copy of the config. Everything else that does affect the embeddings (model file, backend, options, …) still moves the fingerprint as before.

Notes for Reviewers

  • How it was tested: new Ginkgo spec in core/application/router_factories_test.go ("stays stable when a chat request persists probed reasoning slots") — it fails without the change and passes with it; the existing router_factories specs still pass (12/12, ginkgo --focus router_factories ./core/application/). go vet ./core/application/ and gofmt are clean. I did not run golangci-lint or the full make test locally.
  • Seen in production on an AMD Strix Halo appliance: auto routers returned 503 right after someone chatted with the embedding model; a restart cleared it.
  • No breaking change, no config or API change, so no docs update (Docs-Not-Needed in the commit).

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

🤖 Generated with Claude Code

A chat completion sent to an embedding model whose config sets
use_tokenizer_template runs the thinking probe, and persistProbedReasoning
writes the probed reasoning slots back into the shared model config.
EmbedderFingerprint hashed the whole config, so that single request moved
the fingerprint and every knn router built on the embedder failed with
ErrLiveEmbeddingMismatch (HTTP 503) until LocalAI was restarted.

Reasoning settings never change the embedding space, so leave them out of
the hash.

Docs-Not-Needed: internal fingerprint computation, no user-facing config, flag or API change

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

@mudler Code review sign-off at 46163dc1d4742fff3dcc313009d4764fc835f762, subject to CI. The copied config excludes the reasoning fields without mutating the live config; the remaining config and artifact identity checks still participate in the fingerprint. The regression spec covers the two persisted reasoning slots, alongside the existing config/artifact invalidation spec.

Reviewed against current master 6343a2dc6f45d6aae2c98d530b1e6c60deb9b574; the trial merge is clean and git diff --check passes. I could not independently run the focused specs: compilation stops because this worktree's generated protobuf bindings lack Animate3DRequest and Result.Metadata. No passing local-test claim; CI still needs to validate this head.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants