Please provide a way for sysadmins to force use of an HTTP proxy server

[mtdcr]

Is your feature request related to a problem?

Dear everyone,

first let met thank you for creating this awesome library!

As a sysadmin, I would like to force processes using this library to use my proxy server. My goal is to limit outgoing connections to selected hosts, thereby blocking connections to hosts that might be used by malware in case of a compromised host.

Most other HTTP client libraries already support this by respecting environment variables like https_proxy and no_proxy. I noticed that aiohttp already supports these variables in theory, but it requires software developers to opt-in. However, software developers are not the people who know the environments the programs are run in. It is a decision that sysadmins ought to make.

Big and successful Python projects like Home Assistant or Music Assistant include a multitude of libraries embedding aiohttp. It is not feasible to patch each invocation of aiohttp's API to enable the use of proxy servers.

As a result, what I'm doing right now looks like this for a container running in Podman, but is a waste of resources if everybody first has to find out how to do it, because it can't be documented centrally:

FROM ghcr.io/music-assistant/server:latest
RUN ["sh", "-c", "sed -e 's#trust_env: bool = False,#trust_env: bool = True,#' -i /app/venv/lib/python3.*/site-packages/aiohttp/client.py"]

Most people won't know how to inject a patch like that. Most people won't be able to increase security by using a proxy server with applications using aiohttp.

Describe the solution you'd like

Please provide a means to let sysadmins express their wish to use proxy servers.

Options could be (non-exhaustive):

• Just enabling the already known environment variables as default values, if no other proxy settings were set in the API
• Using an additional variable (e. g. AIOHTTP_TRUST_ENV=1)´ to enable the variables for aiohttp.
• Using a configuration file for aiohttp or a flag visible inside the filesystem like ~/.aiohttp_trust_env.

Describe alternatives you've considered

Endless manual patching. ;-)

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[bdraco]

Thanks for the detailed write-up. This is a real pain point and we get the frustration.

The trust_env setting is secure by default and inconvenient for many use cases, and that was deliberate. Functionally, the risk of changing the default is pretty low since if an attacker can control env vars, they can almost certainly compromise the system anyway. But trust_env=False has been the default forever, and some users may rely on it as a security measure. We honestly have no way of knowing how it's used out there, and changing it would be pulling the rug out from under anyone who depends on that behavior.

An env var like AIOHTTP_TRUST_ENV=1 seems like the obvious answer, but the semantics are awful. trust_env=False specifically means "don't trust the environment." Checking an env var to decide whether to trust env vars is contradictory and undermines the whole point.

It's the classic usability vs secure-by-default problem, made worse by backcompat. Changing this would need a major version bump, plenty of advance warning, and clear docs about the security implications. That's unlikely to happen without all of those.

For now, the workarounds are what they are: patching the package like you're doing, asking application maintainers to expose trust_env as a config option, or network-level enforcement with firewall rules or transparent proxies.

None of those are great, but we don't have a clean answer that respects both the security guarantees some users depend on and the flexibility sysadmins need.

[mtdcr]

Thank you for your elaborate response, @bdraco!

I thought about your words for some days and, with my security hat put on, failed to imagine any real threat possibly being caused by respecting proxy environment variables. As you stated, an attacker controlling environment variables basically already controls the whole user account used to execute the application. An attacker with such capabilities will also be able to modify PATH and likely be able to modify the filesystem, as an easy to understand example, and therefore cause much worse problems for users than those implied by the existence of the trust_env parameter. Seeing this parameter as a way to strengthen security feels kind of esoteric to me. This is why I think the current state is not "secure by default" at all. It's just an arbitrarily chosen behavior by default, totally unrelated to security. It may have come from a felt sense of improved security. I recognize, however, that changing the default may break some installations, where invalid proxy settings are present due to misconfiguration.

In contrast, allowing sysadmins (i.e. your users) to use real protection mechanisms like proxy servers (without needing to become developers) produces real, measurable benefits for the general public. Most people won't become developers (and sadly, most people won't have an easy to use filtering proxy server either, but they might find other people to try to help them make their networks more secure). One could say that the current state even makes aiohttp insecure by default.

Firewall rules and transparent proxies are no alternatives: Allowing connections on IP subnet level does not scale in a world in which every other service uses hyperscalers and CDNs. You'd basically end up allowing connections to a huge pile of services never heard of before. Transparent proxying, on the other hand, does not work at all with HTTPS.

I fully understand that backwards compatibility is one of the most important aspects for every library. Still, going forward, creating a major release changing reasonable defaults seems to be the right thing to do.

[mtdcr]

One more addition: Using the HTTP CONNECT request method to establish a TLS connection is secure, even if a user-configured proxy is not trusted by the software developer, because certificate validation still happens the same way on client (and possibly server) as without a proxy. In case you won't be convinced that trust_env is not a security-improving option in itself, you might consider enabling HTTPS connections through proxies using that method by default in a future version of aiohttp. I keep my fingers crossed.