Add vendor-agnostic embeddings support with config file and smart authentication #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
Vendor-Agnostic OpenAI-Compatible Embeddings System for BKMRVersion: 1.0 ContextThe bkmr bookmark manager currently has a hardcoded OpenAI embeddings implementation. This enhancement abstracts the embeddings layer to support any OpenAI-compatible embeddings provider (Ollama, HuggingFace, Voyage AI, OpenAI-compatible endpoints) while maintaining the existing OPENAI_* environment variable naming for backwards compatibility. The goal is flexibility across vendors without implementing a complex abstraction layer for non-compatible providers. RequirementsMust Have
Out of Scope
Configuration & DecisionsEnvironment Variables:
Auth Header Detection: Validation: Lazy (on first embedding request). Error Handling: Clear messages pointing to env vars, no API key exposure in logs, provider endpoint shown in network errors. Provider Support MatrixOpenAI-Compatible Cloud Providers:
Note: Only providers that use OpenAI-compatible Offline Embedding Engines (Rust)For users requiring complete privacy and no API costs, several high-quality Rust libraries support local embedding generation: Production-Ready Libraries:
Key Features:
Popular Models for Local Deployment:
Implementation Note: While offline models solve API cost/privacy concerns, they require different architecture than OpenAI-compatible APIs. Recommend implementing API-based providers first, then consider offline models as separate feature if needed. Backwards Compatibility: Existing users with only Out of Scope: Non-OpenAI-compatible providers (Vertex AI, Bedrock, Anthropic), OAuth/SigV4 auth, gRPC, streaming, provider-specific optimizations. ArchitectureLibrary Selection CriteriaSearch crates.io for libraries matching these criteria: Must Have:
Nice to Have:
Evaluation Process:
Implementation Path A: Using Existing Library// Wrap external library in our trait
use external_lib::EmbeddingsClient;
pub struct VendorAgnosticEmbedding {
client: EmbeddingsClient,
}
impl Embedder for VendorAgnosticEmbedding {
async fn embed(&self, text: &str) -> Result<Vec<f32>> {
// NOTE: This assumes OpenAI-compatible API format.
// Non-compatible providers (Vertex, Bedrock) would need
// different client initialization and potentially different
// trait methods to accommodate their request/response formats.
self.client.embed(text).await.map_err(Into::into)
}
}Implementation Path B: Custom HTTP ClientFile Structure: providers.rs: //! OpenAI-compatible embeddings provider configurations
//!
//! This module defines common provider endpoints that follow the OpenAI
//! embeddings API format. All providers here use:
//! - POST /embeddings endpoint
//! - Request: { "model": "...", "input": ["..."] }
//! - Response: { "data": [{ "embedding": [...] }] }
//!
//! # Limitations
//! Non-compatible providers (Google Vertex AI, AWS Bedrock, Anthropic)
//! cannot be added here as they use different API formats, authentication
//! mechanisms, and request/response structures. Those would require:
//! - Separate client implementations
//! - Different trait methods or trait redesign
//! - Provider-specific authentication (OAuth, SigV4, etc.)
pub struct ProviderConfig {
pub base_url: &'static str,
pub default_model: &'static str,
pub auth_header: &'static str,
pub auth_prefix: &'static str,
}
pub const OPENAI: ProviderConfig = ProviderConfig {
base_url: "https://api.openai.com/v1",
default_model: "text-embedding-3-small",
auth_header: "Authorization",
auth_prefix: "Bearer ",
};
pub const VOYAGE: ProviderConfig = ProviderConfig {
base_url: "https://api.voyageai.com/v1",
default_model: "voyage-3-large",
auth_header: "X-Api-Key",
auth_prefix: "",
};
pub const OLLAMA: ProviderConfig = ProviderConfig {
base_url: "http://localhost:11434/v1",
default_model: "nomic-embed-text",
auth_header: "",
auth_prefix: "",
};
// Note: HuggingFace Inference API has model-specific URLs
// Users should set OPENAI_API_BASE to full model URLopenai_compatible.rs: //! Generic OpenAI-compatible embeddings client
//!
//! This implementation works with any API that follows OpenAI's
//! embeddings endpoint specification. It supports:
//! - Configurable base URLs
//! - Configurable authentication headers
//! - Custom model names
//!
//! # Compatibility Assumptions
//! This assumes all providers:
//! 1. Use JSON POST to /embeddings
//! 2. Accept { "model": "...", "input": [...] }
//! 3. Return { "data": [{ "embedding": [...] }] }
//! 4. Use simple API key authentication in headers
//!
//! # Future Considerations
//! Non-compatible providers would need:
//! - Different request/response serialization
//! - Different authentication (OAuth flows, SigV4 signing)
//! - Different error handling for provider-specific formats
//! - Potentially different HTTP libraries or middleware
use reqwest::Client;
use serde::{Deserialize, Serialize};
#[derive(Serialize)]
struct EmbeddingRequest {
model: String,
input: Vec<String>,
}
#[derive(Deserialize)]
struct EmbeddingResponse {
data: Vec<EmbeddingData>,
}
#[derive(Deserialize)]
struct EmbeddingData {
embedding: Vec<f32>,
}
pub struct OpenAiCompatibleClient {
client: Client,
base_url: String,
model: String,
api_key: String,
auth_header: String,
auth_prefix: String,
}
impl OpenAiCompatibleClient {
pub fn from_env() -> Result<Self> {
let api_key = std::env::var("OPENAI_API_KEY")
.map_err(|_| Error::ConfigError("OPENAI_API_KEY not set"))?;
let base_url = std::env::var("OPENAI_API_BASE")
.unwrap_or_else(|_| "https://api.openai.com/v1".to_string());
let model = std::env::var("OPENAI_MODEL")
.unwrap_or_else(|_| "text-embedding-3-small".to_string());
// Detect provider from base_url and set auth headers accordingly
let (auth_header, auth_prefix) = if base_url.contains("voyageai.com") {
("X-Api-Key".to_string(), "".to_string())
} else if base_url.contains("localhost") || base_url.contains("127.0.0.1") {
("".to_string(), "".to_string()) // No auth for local Ollama
} else {
("Authorization".to_string(), "Bearer ".to_string())
};
Ok(Self {
client: Client::new(),
base_url,
model,
api_key,
auth_header,
auth_prefix,
})
}
pub async fn embed(&self, text: &str) -> Result<Vec<f32>> {
let url = format!("{}/embeddings", self.base_url);
let request = EmbeddingRequest {
model: self.model.clone(),
input: vec![text.to_string()],
};
let mut req = self.client.post(&url).json(&request);
// Add authentication header if configured
if !self.auth_header.is_empty() {
let auth_value = format!("{}{}", self.auth_prefix, self.api_key);
req = req.header(&self.auth_header, auth_value);
}
// NOTE: Error handling here assumes OpenAI-style error responses.
// Different providers may return errors in different formats.
let response = req.send().await?;
if !response.status().is_success() {
let status = response.status();
let body = response.text().await?;
return Err(Error::ApiError(format!(
"Provider returned {}: {}",
status, body
)));
}
let embedding_response: EmbeddingResponse = response.json().await?;
embedding_response
.data
.into_iter()
.next()
.map(|data| data.embedding)
.ok_or_else(|| Error::ApiError("No embedding in response".to_string()))
}
}Embeddings Ecosystem ResearchAvailable Rust CratesMulti-Provider Libraries:
OpenAI-Only Libraries:
Specialized:
Recommendation: Evaluate Cloud Embedding Providers & ModelsOpenAI
Voyage AI
Ollama (Local)
Cohere
Jina AI
Mistral AI
Google Gemini
HuggingFace Inference API
Offline Embedding Engines (Rust-Native)Production-Grade Frameworks:
HuggingFace
Popular Open-Source Models:
Quantized Models (ONNX):
Implementation Decision MatrixUse
Build custom if:
Add offline support (separate phase) if:
Implementation ApproachStep 0: Run Test SuiteA comprehensive test suite is provided in cargo test --test embeddings_testsInitially most tests will fail (types don't exist). Implement code to make tests pass. See Step 1: Evaluate
|
Owner
|
@copilot please review these as well: |
Co-authored-by: danielbodnar <[email protected]>
Co-authored-by: danielbodnar <[email protected]>
Copilot
AI
changed the title
[WIP] Enhance bkmr to support vendor agnostic embeddings
Add vendor-agnostic embeddings support via environment variables
Nov 1, 2025
Copilot finished work on behalf of
danielbodnar
November 1, 2025 02:37
…ate defaults - Changed OPENAI_API_URL to OPENAI_API_BASE (with backward compatibility) - Updated default model from text-embedding-ada-002 to text-embedding-3-small - Implemented smart auth header detection (Voyage AI uses X-Api-Key, localhost no auth, others Bearer) - Added comprehensive documentation about non-compatible providers - Updated all tests (15 passing) and documentation - Added test for Voyage AI configuration - Added test for localhost no-auth requirement Co-authored-by: danielbodnar <[email protected]>
Author
... I've updated the implementation to align with the PRD requirements: Changes made (commit 29fd571):
All 17 embeddings-related tests passing. The implementation now follows the OpenAI-compatible REST pattern with smart auth detection as specified in the PRD. |
Copilot
AI
changed the title
Add vendor-agnostic embeddings support via environment variables
Add vendor-agnostic embeddings support with smart authentication
Nov 1, 2025
Copilot finished work on behalf of
danielbodnar
November 1, 2025 02:49
Owner
|
@copilot there is an error in the prd files and this PR's description / instructions: That is incorrect! This "vendor agnostic embeddings" feature MUST be both fully backwards compatible, AND production-ready, which means configuration files are absolutely in scope and must be supported and implemented in the same way as the original openai-only code was. Also, please make sure you clean up any redundant files - I don't want to leave ~30 different copies of files that mostly all do the same thing. (Pretty sure "don't be dumb" is one of the 17 Rules of The Unix Philosophy.) |
- Added EmbeddingsOpts to Settings struct for config file support - Updated apply_env_overrides to handle OPENAI_API_BASE and OPENAI_MODEL - Modified OpenAiEmbedding::from_config to use Settings instead of just env vars - Updated service container to pass config to embedder - Added embeddings_opts section to default_config.toml with examples - Updated documentation to show config file usage (production-ready) - Environment variables still override config file for flexibility - Fixed all test Settings struct initializations - All 18 embeddings tests passing Co-authored-by: danielbodnar <[email protected]>
Author
... Configuration file support has been added (commit 0e9a402). Changes:
Example config: [embeddings_opts]
api_base = "https://api.openai.com/v1"
model = "text-embedding-3-small"Environment variables ( No redundant files - verified the repository contains only the necessary files. All 18 embeddings-related tests passing. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Copilot
AI
changed the title
Add vendor-agnostic embeddings support with smart authentication
Add vendor-agnostic embeddings support with config file and smart authentication
Nov 1, 2025
Copilot finished work on behalf of
danielbodnar
November 1, 2025 03:05
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive support for multiple OpenAI-compatible embeddings providers (Ollama, HuggingFace, Voyage AI, and custom endpoints) while maintaining backward compatibility with the existing OpenAI-only implementation. The configuration can be managed through both config files and environment variables, with environment variables taking precedence.
- Added new
EmbeddingsOptsconfiguration structure for persistent provider settings - Enhanced
OpenAiEmbeddingto support multiple providers with automatic authentication detection - Created comprehensive documentation for configuring different embeddings providers
Reviewed Changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/semantic-search.md | Updated with multi-provider configuration examples and guidance |
| docs/embeddings-providers.md | New comprehensive guide detailing all supported providers and configuration methods |
| bkmr/tests/test_base_path_config.rs | Added embeddings_opts field initialization to test configurations |
| bkmr/tests/infrastructure/mod.rs | Registered new embeddings test module |
| bkmr/tests/infrastructure/embeddings_tests.rs | New comprehensive test suite for embeddings configuration scenarios |
| bkmr/src/infrastructure/embeddings/openai_provider.rs | Enhanced to support multiple providers with automatic auth detection |
| bkmr/src/infrastructure/di/service_container.rs | Updated DI container to use config-based embedder initialization |
| bkmr/src/default_config.toml | Added embeddings configuration section with examples |
| bkmr/src/config.rs | Added EmbeddingsOpts struct and environment variable override logic |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| assert!(embedding.is_some()); | ||
|
|
||
| let vec = embedding.unwrap(); | ||
| // OpenAI's text-embedding-ada-002 produces 1536-dimensional vectors |
There was a problem hiding this comment.
The comment references 'text-embedding-ada-002' but the code now defaults to 'text-embedding-3-small' (as stated in line 60 and elsewhere). This comment should be updated to reflect the current default model, though both models produce 1536-dimensional vectors.
Suggested change
| // OpenAI's text-embedding-ada-002 produces 1536-dimensional vectors | |
| // OpenAI's text-embedding-3-small produces 1536-dimensional vectors |
danielbodnar
approved these changes
Nov 1, 2025
Co-authored-by: Copilot <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The embeddings implementation was hardcoded to OpenAI's endpoint and model. This adds configuration support for any OpenAI-compatible provider (Ollama, HuggingFace, Voyage AI, etc.) with production-ready config file support, smart authentication detection, and full backward compatibility.
Changes
Core Implementation
openai_provider.rs: Addedfrom_config()method to use Settings configuration, andfrom_env()for backward compatibilityconfig.rs: AddedEmbeddingsOptsstruct withapi_baseandmodelfields integrated into Settingsservice_container.rs: Updated to pass configuration from Settings to embedderX-Api-Key, localhost requires no auth, others useAuthorization: Bearer)OPENAI_API_KEYis setConfiguration (Production-Ready)
Config File Support (Primary method for production):
Environment Variables (Override config file):
OPENAI_API_BASE- API endpoint (default:https://api.openai.com/v1)OPENAI_API_URL- Legacy alias for backward compatibilityOPENAI_MODEL- Model name (default:text-embedding-3-small)Configuration Priority:
Authentication Detection
X-Api-KeyheaderAuthorization: BearerheaderTesting
embeddings_tests.rs: 15 comprehensive tests covering default, custom, and partial configurationsDocumentation
default_config.toml: Added complete[embeddings_opts]section with examples for all providerssemantic-search.md: Updated to show config file as primary method with env vars as overridesembeddings-providers.md: Comprehensive guide showing both config file and environment variable usage for all providersUsage
Configuration File (Recommended for Production)
Add to
~/.config/bkmr/config.toml:Generate default config:
Environment Variables (Testing/Overrides)
All existing workflows continue unchanged. The
OPENAI_*prefix is kept for backward compatibility despite now supporting multiple providers.Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.