Resolve test case [It] [sig-cli] Kubectl client Simple pod should support exec through an HTTP proxy

[BenTheElder]

[It] [sig-cli] Kubectl client Simple pod should support exec through an HTTP proxy is one of the few (only?) test cases that would still fail with the straightforward test filter Feature: isEmpty && !Slow && !Disruptive && !Flaky (+ skip regex = \[Serial\] which we inject when running tests in parallel, such as presubmit)

This test fails on kind. We should determine if this is a bug in the test, kind, the test setup ...

It seems to rarely fail in few other jobs, so I'm not default assuming it's the test

https://storage.googleapis.com/k8s-triage/index.html?pr=1&test=should%20support%20exec%20through%20an%20HTTP%20proxy

We have been able to eliminate almost all of the other skips, e.g. via NFS now working in kind. See: https://github.com/kubernetes/test-infra/pull/34600/files

/sig testing cli
cc @aojea @soltysh @eddiezane

[BenTheElder]

Example failure: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kind/648/pull-kind-e2e-kubernetes-canary/1904299874459848704

From: #648 (test / empty PR)

{ failed [FAILED] Missing expected log result on proxy server for https_proxy. Expected: "Accepting CONNECT to 127.0.0.1:34867", got ""
In [It] at: k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:499 @ 03/24/25 22:51:12.085
}

[BenTheElder]

Might be mixing external (e2e.test) / internal (pod) API server addresses?

[BenTheElder]

[It] [sig-cli] Kubectl client Simple pod should support exec through an HTTP proxy is one of the few (only?) test cases that would still fail with the straightforward test filter Feature: isEmtpy && !Slow && !Disruptive && !Flaky (+ skip regex = [Serial] which we inject when running tests in parallel, such as presubmit)

Yes, the only after kubernetes/test-infra#34600

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kind/648/pull-kind-e2e-kubernetes-canary/1904319372680237056

[BenTheElder]

After some debugging ... yeah so kubectl genuinely does not use the proxy when the cluster server address is on localhost. (or in our case 127.0.0.1)

Turns out, this has been reported before kubernetes/kubectl#743 (comment)

https://github.com/golang/net/blob/ebed060e8f30f20235f74808c22125fd86b15edd/http/httpproxy/proxy.go#L185

I guess the test could self-skip if on localhost / loopback ...? Hmmm

[BenTheElder]

Debugging improvements in the meantime: kubernetes/kubernetes#131060

[BenTheElder]

After some debugging ... yeah so kubectl genuinely does not use the proxy when the cluster server address is on localhost. (or in our case 127.0.0.1)

The kind server address is of course on localhost loopback.

We can make it not on localhost, but that's a pain.
We either need to require messing with hostname resolution, or we have to run the tests inside the container network.

I've been tempted by the latter for a while for other reasons, but it's a messy change that needs some thought, and I wouldn't want to rush it for this test.

Unless we break the behavior, this test can't apply to clusters on loopback/localhost, and we're probably not the only ones doing that.

Maybe we should sniff the server address and self-skip ...

[BenTheElder]

We could make the test use an http_proxy running inside the cluster and run kubectl there? 🤔

[BenTheElder]

Ideas:

1. run kubectl + proxy in cluster and manipulate /etc/hosts in the kubectl pod or use a in-cluster service domain for the proxy URL ... little messy, doesn't test the local kubectl binary, but it would be portable.
2. have the test self-skip if cluster server is on loopback/localhost
3. continue specially excluding in kind job configs
4. alter kubectl behavior to proxy anyhow unless NO_PROXY is set .... sounds pretty iffy
5. require running tests on a host that has a non-localhost /etc/hosts entry for the kind cluster .... not portable, host global ...

Thoughts, @soltysh @eddiezane @aojea @pohly ?

[BenTheElder]

1. seems bad because there's no reasonable way to guarantee we test the kubectl we just built, short of like, self-copying kubectl into the pod 🤯 , which sounds like a neat trick and probably very doable but hard to seriously suggest.

2. could be OK, I guess.

3. I would prefer to avoid, we want to rely on some mechanism like test tagging, self-skipping etc instead, so that in the future the test can be fixed/updated without "skipping based on regex of test name" tripping us up. (e.g. !Slow leaves room to send a PR to k/k to remove the slow tag)

4. ... I fear the breakage we'd induce, though it could be mitigated by NO_PROXY, the downside to NOT doing 4 is that you don't appear to be able to force kubectl to use a proxy for a local address without some hackery (like 1 / kubectl doesn't use specified HTTP/HTTPS proxy if server is localhost kubernetes/kubectl#743 (comment)), but the use case for that has to be rather niche?

5. possible, though the test would still be confusing if anyone ever runs it from a debugger locally. seems fragile.

[aojea]

Setting proxy-url on the config solves the problem of the interpretation of the environment variables on the std golang library, example

Start proxy listening on localhost:8888 (default)

go run github.com/jthomperoo/simple-proxy@main
go: downloading github.com/jthomperoo/simple-proxy v1.3.0
go: downloading github.com/golang/glog v1.0.0
I0327 11:07:56.257407 3973275 main.go:117] Starting HTTP proxy...

Run Pod

 kubectl run test --image=httpd:2

exec without proxy

kubectl exec test -- echo "hello"
hello

set the config to us a proxy

kubectl config set clusters.kind-kind.proxy-url http://localhost:8888

exec with proxy

$ http_proxy=:8888 https_proxy=:8888 kubectl -v 7 exec test -- echo "hello"
I0327 11:27:17.878645 3990380 loader.go:402] Config loaded from file:  /usr/local/google/home/aojea/.kube/config
I0327 11:27:17.879264 3990380 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0327 11:27:17.879281 3990380 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0327 11:27:17.879285 3990380 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0327 11:27:17.879290 3990380 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0327 11:27:17.882769 3990380 round_trippers.go:470] GET https://127.0.0.1:42661/api/v1/namespaces/default/pods/test
I0327 11:27:17.882787 3990380 round_trippers.go:476] Request Headers:
I0327 11:27:17.882794 3990380 round_trippers.go:480]     User-Agent: kubectl/v1.32.2 (linux/amd64) kubernetes/67a30c0
I0327 11:27:17.882801 3990380 round_trippers.go:480]     Accept: application/json, */*
I0327 11:27:17.889048 3990380 round_trippers.go:581] Response Status: 200 OK in 6 milliseconds
I0327 11:27:17.891324 3990380 podcmd.go:88] Defaulting container name to test
I0327 11:27:17.891635 3990380 round_trippers.go:470] GET https://127.0.0.1:42661/api/v1/namespaces/default/pods/test/exec?command=echo&command=hello&container=test&stderr=true&stdout=true
I0327 11:27:17.891646 3990380 round_trippers.go:476] Request Headers:
I0327 11:27:17.891653 3990380 round_trippers.go:480]     User-Agent: kubectl/v1.32.2 (linux/amd64) kubernetes/67a30c0
I0327 11:27:17.891660 3990380 round_trippers.go:480]     Sec-Websocket-Protocol: v5.channel.k8s.io
I0327 11:27:17.897432 3990380 round_trippers.go:581] Response Status: 101 Switching Protocols in 5 milliseconds
I0327 11:27:17.897448 3990380 websocket.go:133] The subprotocol is v5.channel.k8s.io

The proxy logs the request

I0327 11:28:07.417159 3977563 proxy.go:37] Serving 'CONNECT' request from '[::1]:57484' to '127.0.0.1:42661'
I0327 11:28:07.474394 3977563 proxy.go:37] Serving 'CONNECT' request from '[::1]:57498' to '127.0.0.1:42661'
I0327 11:28:07.482759 3977563 proxy.go:37] Serving 'CONNECT' request from '[::1]:57514' to '127.0.0.1:42661'

Another think that will work on hosts that resolve the hostname is to replace the server by the hostname, but this require to obtain the existing endpoint

http_proxy=:8888 https_proxy=:8888 kubectl -v 7 --server $(hostname):PORT??? exec test -- echo "hello"

[BenTheElder]

Setting proxy-url on the config solves the problem of the interpretation of the environment variables on the std golang library, example

This also works as a flag, right? Either way, the tests could generate a temporary config copy with this set.

I think it depends on if sig-cli is OK with testing just this way, or trying specifically to test the env(s).

https://kubernetes.slack.com/archives/C09QZ4DQB/p1743098360811159?thread_ts=1742942034.393419&cid=C09QZ4DQB

Another think that will work on hosts that resolve the hostname is to replace the server by the hostname, but this require to obtain the existing endpoint

Yeah, that's 5) which I don't think we should do.