fix: astream is not sending token in python 3.10 #33907
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In Python 3.10, asynchronous generators may not yield control immediately after
yield, causing the event loop to be unable to process other tasks in a timely manner and thus affecting the real-time performance of streaming transmission. Python 3.12 has optimized the behavior of asynchronous generators, so they function properly.When calling agent.astream() or similar asynchronous streaming methods, you won't receive each token in real-time. All tokens are returned at once after the message is fully generated. In Python 3.12, the same code can properly receive tokens one by one.
The solution is to add
await asyncio.sleep(0)after eachyieldstatement in the asynchronous generator to explicitly yield control, ensuring that the event loop can process streaming data in a timely manner.