fix: Extract usage info from any chunk in streaming response #10265
Conversation
The previous implementation only looked at the last chunk for usage info, which caused issues with some API providers (like Qwen3) that may return usage in a different chunk than the final one. Now we iterate through chunks in reverse order to find the last non-None usage value, making usage extraction more robust across different providers. Closes deepset-ai#10258
|
@majiayu000 is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
|
|
Hey @majiayu000 this is actually a good fix, please read https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md and update your PR |
|
Thanks for the feedback! I've added the release notes. Ready for review. |
Nice, couple of formatting issues still - see the failures, you can reproduce them locally - all commands on CI are available locally as well |
Pull Request Test Coverage Report for Build 20404913201Details
💛 - Coveralls |
| usage = next((u for chunk in reversed(chunks) if (u := chunk.meta.get("usage")) is not None), None) | ||
|
|
||
| meta = { | ||
| "model": chunks[-1].meta.get("model"), |
There was a problem hiding this comment.
Does getting a model and received_at work for Qwen @majiayu000 ?
There was a problem hiding this comment.
Hi @vblagoje, I tested with Qwen3 (qwen/qwen3-235b-a22b) via OpenRouter:
modelandreceived_atappear in every chunk, so getting them fromchunks[-1]works correctlyusageonly appears in the last chunk
The current implementation should work fine for these fields.
Note: This was tested via OpenRouter. If you have access to Qwen's native DashScope API, additional verification would be helpful as the original issue was reported with the native API.
majiayu000
force-pushed
the
fix/issue-10258-qwen3-streaming-usage
branch
from
cef261d to
d0532cd
Compare
|
Thanks @majiayu000 - I'll take a look today one last time and we'll move forward to integrate! |
4 participants
Summary
Fixed usage info extraction in streaming responses to search all chunks instead of only the last one
This fixes issues with providers like Qwen3 that return usage in a different chunk than the final one
fixes The Qwen3 model cannot obtain the meta usage for streaming. #10258
Test plan
Closes #10258