Skip to content

fix(azure): return embedding token usage instead of only logging it - #2224

Closed
bobmcwhirter wants to merge 2 commits into
0xPlaygrounds:mainfrom
CoreStory-Frontier:upstream-azure-embedding-usage
Closed

fix(azure): return embedding token usage instead of only logging it#2224
bobmcwhirter wants to merge 2 commits into
0xPlaygrounds:mainfrom
CoreStory-Frontier:upstream-azure-embedding-usage

Conversation

@bobmcwhirter

Copy link
Copy Markdown

Azure's embedding client reports zero token usage to callers, because it never overrides EmbeddingModel::embed_texts_with_usage.

The data isn't missing — it's discarded. embed_texts already parses response.usage, hands it to tracing::info!, and then drops it on the floor:

// crates/rig-core/src/providers/azure.rs
ApiResponse::Ok(response) => {
    tracing::info!(target: "rig", "Azure embedding token usage: {}", response.usage);
    ...
    Ok(response.data.into_iter()...collect())   // usage gone
}

Since the trait's default embed_texts_with_usage returns Usage::default() (embeddings/embedding.rs), anything reading EmbeddingResponse::usage gets zeroes for every Azure embedding request.

The change

Restructured exactly the way providers/openai/embedding.rs already does it: embed_texts_with_usage does the work and returns the usage, embed_texts delegates and drops it. 14 insertions, 2 deletions, one file.

No new mapping code was needed — impl GetTokenUsage for Usage already exists in azure.rs and was simply unused for embeddings. The tracing::info! line is retained, so anything watching logs is unaffected.

How it was found, and verified

Measured against a real Azure OpenAI deployment: 100 embedding requests, 0 tokens recorded. After this change the same workload reports the provider's actual prompt_tokens. In our case it meant every embedding cost figure downstream read $0.00 on real spend, which is how we noticed.

Test

An #[ignore]d live test (tests/providers/azure/embeddings.rs), which is this provider's existing convention — there is no tests/cassettes/azure/ fixture set or with_azure_cassette helper to extend, and both current azure tests are #[ignore]-gated on AZURE_OPENAI_API_KEY. CONTRIBUTING lists ignored live tests as acceptable where cassette replay is unsuitable.

I deliberately did not hand-author or record a cassette: recording one against our deployment would have committed our Azure resource and deployment identifiers into this repository. Happy to add a cassette if you'd prefer one recorded from your own fixtures — say the word and I'll follow up.

cargo fmt --check clean; cargo check -p rig-core and cargo test --test azure --no-run both build.

Related, but deliberately out of scope

Only 2 of the 8 provider files implementing embed_texts override embed_texts_with_usage (openai, voyageai). The other six inherit the zero-returning default, so they very likely share this defect — I've kept this PR to azure to stay small and reviewable per CONTRIBUTING, rather than touching providers I can't test.

Worth considering separately: a default implementation that silently returns plausible-but-wrong zeros is easy to inherit by accident, and it took a production cost discrepancy for us to notice. Making embed_texts_with_usage a required method, or having the default return something callers can distinguish from a real measurement, would surface this class of bug at compile time instead. Glad to open an issue for that discussion if it's useful.

Bob McWhirter added 2 commits July 28, 2026 14:55
`EmbeddingModel::embed_texts_with_usage` has a default body that discards usage
and returns `Usage::default()` (embeddings/embedding.rs). Azure's embedding
client never overrode it, so every Azure embedding call reports zero tokens to
any caller reading `EmbeddingResponse::usage` -- while the data was right there:
`embed_texts` parsed `response.usage` and passed it to `tracing::info!`, then
threw it away.

Restructure the impl the way providers/openai/embedding.rs already does:
`embed_texts_with_usage` does the work and returns the usage, `embed_texts`
delegates and drops it. The mapping needed no new code -- `impl GetTokenUsage
for Usage` already existed in this file and was simply unused for embeddings.

Measured against a real Azure deployment before the change: 100 embedding
requests, 0 tokens recorded. The `tracing::info!` line is kept, so nothing that
watched the log loses anything.

Only 2 of 8 provider files override this method (openai, voyageai), so other
providers likely share the defect; this change is deliberately scoped to azure.
An ignored live test, matching this provider's existing convention -- azure has
no tests/cassettes/azure/ fixture set or helper, and its two existing tests are
both #[ignore]-gated on AZURE_OPENAI_API_KEY.
@gold-silver-copper

Copy link
Copy Markdown
Contributor

Provider pipelines have been greatly consolidated, this should be fixed on main now. Please open a new issue if the bug persists. Thank you 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants