Skip to content

Identify and close single connection in AsyncClient connection pool. #3638

@BlackCloud37

Description

@BlackCloud37

Hello,

I'm trying to use httpx(in asyncio) for my web scraper program.

In my use case, for preventing my connection from banning, I usually want to limit the requests on single connection, which means:

  1. Close a connection after N requests.
  2. Close a connection if it's banned, for example, received HTTP code 429 or any invalid response.

I think both of them can be implemented from getting a connection object from a request, mapping it to it's custom state, and call aclose on it whenever needed.

But by reading the document, I cannot find any methods that I can:

  1. Identify each request's underlying connection, so I can count requests on certain connection.
  2. Close this single connection if satisfied any condition above.

So I currently implement this in "one connection per async client" way. I limited the max_connections to 1 for my client, and each time I want to drop a connection, I throw the client away and create a new one.

As the document says, it's a bad idea to do so. So is there any way I can reuse the clients for my purpose?

Any help would be much appreciated!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions