[e2e test] Cluster API working on self-hosted clusters using ClusterClass failing with timed out error

[arshadd-b]

Which jobs are flaking?

https://testgrid.k8s.io/cluster-api-core-main#capi-e2e-latestk8s-main
https://testgrid.k8s.io/cluster-api-core-main#capi-e2e-main
https://testgrid.k8s.io/cluster-api-core-main#capi-e2e-mink8s-main

Which tests are flaking?

When testing Cluster API working on self-hosted clusters using ClusterClass [ClusterClass] Should pivot the bootstrap cluster to a self-hosted cluster [ClusterClass]

Since when has it been flaking?

Started failing since 10/10/2025

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-cluster-api-e2e-main/1976359094839152640

Testgrid link

No response

Reason for failure (if possible)

No response

Anything else we need to know?

Dashboard link : https://storage.googleapis.com/k8s-triage/index.html?job=.*cluster-api.*(test%7Ce2e)*main&test=When%20testing%20Cluster%20API%20working%20on%20self-hosted%20clusters%20using%20ClusterClass&xjob=.*-provider-.*

Label(s) to be applied

/kind flake
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[sbueringer]

/triage accepted

[arshadd-b]

Trying to debug it further, going through the logs

[arshadd-b]

I had checked the logs
It is failing for MD, MP, control-plane machines

I see these changes were added newly https://github.com/kubernetes-sigs/cluster-api/pull/12726/files#diff-ec1d56d38af5017e559182ed7f7d39969f98d3e9dae6d26d4505fc7aee2dfe64R376 and this test is flaky from August

@sbueringer do you think we should have similar changes like this #12833 ?

[sbueringer]

these changes

Which changes are you referring to?

[arshadd-b]

Which changes are you referring to?

https://github.com/kubernetes-sigs/cluster-api/pull/12726/files#diff-ec1d56d38af5017e559182ed7f7d39969f98d3e9dae6d26d4505fc7aee2dfe64R377

in self_hosted.go we have this change , Earlier we did not had control plane and MD passed here

WaitForControlPlaneToBeUpgraded:      input.E2EConfig.GetIntervals(specName, "wait-control-plane-upgrade"),
				WaitForMachineDeploymentToBeUpgraded: input.E2EConfig.GetIntervals(specName, "wait-machine-deployment-upgrade"),
				WaitForMachinePoolToBeUpgraded:       input.E2EConfig.GetIntervals(specName, "wait-machine-pool-upgrade"),
				WaitForKubeProxyUpgrade:              input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
				WaitForDNSUpgrade:                    input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
				WaitForEtcdUpgrade:                   input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),

[chrischdi]

Analyzed: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-cluster-api-e2e-latestk8s-main/1980591354589220864

• 2/3 CP Machines finished their upgrade
• 1/3 CP Machines stuck with CNI not yet ready: https://storage.googleapis.com/kubernetes-ci-logs/logs/periodic-cluster-api-e2e-latestk8s-main/1980591354589220864/artifacts/clusters/self-hosted-csednw/resources/self-hosted-ua80ju/Machine/self-hosted-csednw-74272-2b762.yaml

Timings for the not yet finished CP Machine self-hosted-csednw-74272-2b762

• 2025-10-21T12:11:07Z CreationTimestamp
• 2025-10-21T12:12:34Z CNI/Kindnet Pod creationTimestamp (pod)
• 2025-10-21T12:12:35Z LastTransitionTime for Ready
• 2025-10-21T12:22:46Z CNI/Kindnet Pod condition Ready=true
• 10/21/25 12:23:05.991 Test failed
• 2025-10-21T12:23:10Z Node condition Ready turns true

Digging further into logs, it turns out that pulling the kindnet image took 10m5.527686443s which should have caused the timeout:

Oct 21 12:22:44.382468 self-hosted-csednw-74272-2b762 containerd[126]: time="2025-10-21T12:22:44.382052964Z" level=info msg="Pulled image \"kindest/kindnetd:v20250214-acbabc1a\" with image id \"sha256:df3849d954c98a7162c7bee7313ece357606e313d98ebd68b7aac5e961b1156f\", repo tag \"docker.io/kindest/kindnetd:v20250214-acbabc1a\", repo digest \"docker.io/kindest/kindnetd@sha256:e3c42406b0806c1f7e8a66838377936cbd2cdfd94d9b26a3eefedada8713d495\", size \"38996835\" in 10m5.527686443s"

source: https://storage.googleapis.com/kubernetes-ci-logs/logs/periodic-cluster-api-e2e-latestk8s-main/1980591354589220864/artifacts/clusters/self-hosted-csednw/machines/self-hosted-csednw-74272-2b762/containerd.log

---

Taking a look at the DockerMachine, we can see that the kindnet image is not part of preLoadImages:

https://storage.googleapis.com/kubernetes-ci-logs/logs/periodic-cluster-api-e2e-latestk8s-main/1980591354589220864/artifacts/clusters/self-hosted-csednw/resources/self-hosted-ua80ju/DockerMachine/self-hosted-csednw-74272-2b762.yaml

So adding it there properly should resolve the case of too long image pull here hopefully.

[chrischdi]

Let's see how much #12907 fixes after some time.

It should be easy to verify if a failure is affected by checking the containerd logs of the failing node if CNI pulling took long. All these cases should not appear anymore.