fix(azure): return embedding token usage instead of only logging it - #2224
Closed
bobmcwhirter wants to merge 2 commits into
Closed
fix(azure): return embedding token usage instead of only logging it#2224bobmcwhirter wants to merge 2 commits into
bobmcwhirter wants to merge 2 commits into
Conversation
added 2 commits
July 28, 2026 14:55
`EmbeddingModel::embed_texts_with_usage` has a default body that discards usage and returns `Usage::default()` (embeddings/embedding.rs). Azure's embedding client never overrode it, so every Azure embedding call reports zero tokens to any caller reading `EmbeddingResponse::usage` -- while the data was right there: `embed_texts` parsed `response.usage` and passed it to `tracing::info!`, then threw it away. Restructure the impl the way providers/openai/embedding.rs already does: `embed_texts_with_usage` does the work and returns the usage, `embed_texts` delegates and drops it. The mapping needed no new code -- `impl GetTokenUsage for Usage` already existed in this file and was simply unused for embeddings. Measured against a real Azure deployment before the change: 100 embedding requests, 0 tokens recorded. The `tracing::info!` line is kept, so nothing that watched the log loses anything. Only 2 of 8 provider files override this method (openai, voyageai), so other providers likely share the defect; this change is deliberately scoped to azure.
An ignored live test, matching this provider's existing convention -- azure has no tests/cassettes/azure/ fixture set or helper, and its two existing tests are both #[ignore]-gated on AZURE_OPENAI_API_KEY.
Contributor
|
Provider pipelines have been greatly consolidated, this should be fixed on main now. Please open a new issue if the bug persists. Thank you 🙏 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Azure's embedding client reports zero token usage to callers, because it never overrides
EmbeddingModel::embed_texts_with_usage.The data isn't missing — it's discarded.
embed_textsalready parsesresponse.usage, hands it totracing::info!, and then drops it on the floor:Since the trait's default
embed_texts_with_usagereturnsUsage::default()(embeddings/embedding.rs), anything readingEmbeddingResponse::usagegets zeroes for every Azure embedding request.The change
Restructured exactly the way
providers/openai/embedding.rsalready does it:embed_texts_with_usagedoes the work and returns the usage,embed_textsdelegates and drops it. 14 insertions, 2 deletions, one file.No new mapping code was needed —
impl GetTokenUsage for Usagealready exists inazure.rsand was simply unused for embeddings. Thetracing::info!line is retained, so anything watching logs is unaffected.How it was found, and verified
Measured against a real Azure OpenAI deployment: 100 embedding requests, 0 tokens recorded. After this change the same workload reports the provider's actual
prompt_tokens. In our case it meant every embedding cost figure downstream read$0.00on real spend, which is how we noticed.Test
An
#[ignore]d live test (tests/providers/azure/embeddings.rs), which is this provider's existing convention — there is notests/cassettes/azure/fixture set orwith_azure_cassettehelper to extend, and both current azure tests are#[ignore]-gated onAZURE_OPENAI_API_KEY. CONTRIBUTING lists ignored live tests as acceptable where cassette replay is unsuitable.I deliberately did not hand-author or record a cassette: recording one against our deployment would have committed our Azure resource and deployment identifiers into this repository. Happy to add a cassette if you'd prefer one recorded from your own fixtures — say the word and I'll follow up.
cargo fmt --checkclean;cargo check -p rig-coreandcargo test --test azure --no-runboth build.Related, but deliberately out of scope
Only 2 of the 8 provider files implementing
embed_textsoverrideembed_texts_with_usage(openai,voyageai). The other six inherit the zero-returning default, so they very likely share this defect — I've kept this PR to azure to stay small and reviewable per CONTRIBUTING, rather than touching providers I can't test.Worth considering separately: a default implementation that silently returns plausible-but-wrong zeros is easy to inherit by accident, and it took a production cost discrepancy for us to notice. Making
embed_texts_with_usagea required method, or having the default return something callers can distinguish from a real measurement, would surface this class of bug at compile time instead. Glad to open an issue for that discussion if it's useful.