-
-
Notifications
You must be signed in to change notification settings - Fork 975
Description
Hello,
I'm trying to use httpx(in asyncio) for my web scraper program.
In my use case, for preventing my connection from banning, I usually want to limit the requests on single connection, which means:
- Close a connection after N requests.
- Close a connection if it's banned, for example, received HTTP code 429 or any invalid response.
I think both of them can be implemented from getting a connection object from a request, mapping it to it's custom state, and call aclose on it whenever needed.
But by reading the document, I cannot find any methods that I can:
- Identify each request's underlying connection, so I can count requests on certain connection.
- Close this single connection if satisfied any condition above.
So I currently implement this in "one connection per async client" way. I limited the max_connections to 1 for my client, and each time I want to drop a connection, I throw the client away and create a new one.
As the document says, it's a bad idea to do so. So is there any way I can reuse the clients for my purpose?
Any help would be much appreciated!