Skip to content

Conversation

@nirs
Copy link
Contributor

@nirs nirs commented Dec 10, 2025

We still use go 1.23.4 which fail to buid runc when updating nerdctl to
latest stable version[1]:

go build -trimpath "-buildmode=pie"  -tags "seccomp" -ldflags "-X main.gitCommit=8bd78a9977e604c4d5f67a7415d7b8b8c109cdc4  " -o runc .
# github.com/opencontainers/runc/libcontainer/configs
libcontainer/configs/config.go:332:25: undefined: strings.SplitSeq

strings.SplitSeq was added in go 1.24.0[2]

We use the same GO_VERSION variable for the iso build. This overrides
GO_VERSION in the buildroot go package.

Testing shows that go is installed from binary release, so we have now
checksums for both source and binary releases for amd64 and arm64.
Buildroot builtin go.hash does the same[3].

The golang_version tool need to keep both source a binary releases hashes now (#22106).

[1] https://storage.cloud.google.com/minikube-builds/logs/22099/0a27ed4/iso_build.txt
[2] https://pkg.go.dev/strings#SplitSeq
[3] https://github.com/buildroot/buildroot/blob/master/package/go/go.hash

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Dec 10, 2025
@nirs
Copy link
Contributor Author

nirs commented Dec 10, 2025

/cc @medyagh
/cc @afbjorklund

@nirs
Copy link
Contributor Author

nirs commented Dec 10, 2025

/ok-to-build-iso

@nirs
Copy link
Contributor Author

nirs commented Dec 10, 2025

/ok-to-build-iso

@minikube-bot
Copy link
Collaborator

Hi @nirs, building a new ISO failed for Commit 6836e9f
See the logs at:
https://storage.cloud.google.com/minikube-builds/logs/22101/6836e9f/iso_build.txt

@nirs
Copy link
Contributor Author

nirs commented Dec 11, 2025

The change does not work, we still install go1.23.4:

host-go-bin Downloading
wget -nd -t 3 --connect-timeout=10 -O '/home/jenkins/workspace/iso-pr-build/out/buildroot/output-aarch64/build/.go1.23.4.linux-amd64.tar.gz.CraCW4/output' 'https://go.dev/dl/go1.23.4.linux-amd64.tar.gz'

@afbjorklund any idea what is missing?

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 11, 2025

It seems there is a separate line for the ISO:

# used by hack/jenkins/release_build_and_upload.sh, see also BUILD_IMAGE below
# update this only by running `make update-golang-version`
GO_VERSION ?= 1.25.5
# the go version on the line below is for the ISO
GOLANG_OPTIONS = GOWORK=off GO_VERSION=1.23.4 GO_HASH_FILE=$(PWD)/deploy/iso/minikube-iso/go.hash
BUILDROOT_OPTIONS = BR2_EXTERNAL=../../deploy/iso/minikube-iso $(GOLANG_OPTIONS)

I don't think the "go.mk" is used anywhere?

In theory the above variable would be enough to override the hash file, but in practice there is no such variable...
So we just patch the existing one: cp deploy/iso/minikube-iso/go.hash $(BUILD_DIR)/buildroot/package/go/go.hash

Probably should be upstreamed properly...

The real files for the "go" package are in https://github.com/buildroot/buildroot/tree/2025.02.x/package/go
It is currently using go1.23, which is not new enough to build everything it seems (some even require go1.25)

@nirs nirs changed the title iso: Update go to 1.25.5 iso: Use same Go version (1.25.5) Dec 11, 2025
@nirs
Copy link
Contributor Author

nirs commented Dec 11, 2025

/ok-to-build-iso

@nirs
Copy link
Contributor Author

nirs commented Dec 11, 2025

It seems there is a separate line for the ISO:

# used by hack/jenkins/release_build_and_upload.sh, see also BUILD_IMAGE below
# update this only by running `make update-golang-version`
GO_VERSION ?= 1.25.5
# the go version on the line below is for the ISO
GOLANG_OPTIONS = GOWORK=off GO_VERSION=1.23.4 GO_HASH_FILE=$(PWD)/deploy/iso/minikube-iso/go.hash
BUILDROOT_OPTIONS = BR2_EXTERNAL=../../deploy/iso/minikube-iso $(GOLANG_OPTIONS)

Thanks, I removed the overrides files and just updated the GOLANG_OPTIONS with same GO_VERSION we use elsewhere.

I don't think the "go.mk" is used anywhere?

In theory the above variable would be enough to override the hash file, but in practice there is no such variable... So we just patch the existing one: cp deploy/iso/minikube-iso/go.hash $(BUILD_DIR)/buildroot/package/go/go.hash

Probably should be upstreamed properly...

Trying to make it work first

@minikube-bot
Copy link
Collaborator

Hi @nirs, building a new ISO failed for Commit 98550b3
See the logs at:
https://storage.cloud.google.com/minikube-builds/logs/22101/98550b3/iso_build.txt

@afbjorklund
Copy link
Collaborator

You still need to update the go.hash file, though.

@nirs
Copy link
Contributor Author

nirs commented Dec 11, 2025

You still need to update the go.hash file, though.

The hash faile already have go 1.25.5:

sha256  22a5fd0a91efcd28a1b0537106b9959b2804b61f59c3758b51e8e5429c1a954f  go1.25.5.src.tar.gz

But the build downloads the go binary release:

>>> host-go-bin  Downloading
wget -nd -t 3 --connect-timeout=10 -O '/home/jenkins/workspace/iso-pr-build/out/buildroot/output-aarch64/build/.go1.25.5.linux-amd64.tar.gz.djtEEx/output' 'https://go.dev/dl/go1.25.5.linux-amd64.tar.gz' 
--2025-12-11 15:31:44--  https://go.dev/dl/go1.25.5.linux-amd64.tar.gz
Resolving go.dev (go.dev)... 216.239.32.21, 216.239.36.21, 216.239.34.21, ...
Connecting to go.dev (go.dev)|216.239.32.21|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://dl.google.com/go/go1.25.5.linux-amd64.tar.gz [following]
--2025-12-11 15:31:44--  https://dl.google.com/go/go1.25.5.linux-amd64.tar.gz
Resolving dl.google.com (dl.google.com)... 173.194.193.91, 173.194.193.93, 173.194.193.190, ...
Connecting to dl.google.com (dl.google.com)|173.194.193.91|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 59768009 (57M) [application/x-gzip]
Saving to: ‘/home/jenkins/workspace/iso-pr-build/out/buildroot/output-aarch64/build/.go1.25.5.linux-amd64.tar.gz.djtEEx/output’
...
2025-12-11 15:31:45 (169 MB/s) - ‘/home/jenkins/workspace/iso-pr-build/out/buildroot/output-aarch64/build/.go1.25.5.linux-amd64.tar.gz.djtEEx/output’ saved [59768009/59768009]

ERROR: No hash found for go1.25.5.linux-amd64.tar.gz

Maybe they use binary release if available, and build from source if not?

The go.hash in buildroot includes both source and binary hashes:
https://github.com/buildroot/buildroot/blob/master/package/go/go.hash

@afbjorklund I guess we should do the same?

@afbjorklund
Copy link
Collaborator

There should be a config for that:

BR2_PACKAGE_HOST_GO_SRC
BR2_PACKAGE_HOST_GO_BIN

But then you need more checksums?

We still use go 1.23.4 which fail to buid runc when updating nerdctl to
latest stable version[1]:

    go build -trimpath "-buildmode=pie"  -tags "seccomp" -ldflags "-X main.gitCommit=8bd78a9977e604c4d5f67a7415d7b8b8c109cdc4  " -o runc .
    # github.com/opencontainers/runc/libcontainer/configs
    libcontainer/configs/config.go:332:25: undefined: strings.SplitSeq

strings.SplitSeq was added in go 1.24.0[2]

We use the same GO_VERSION variable for the iso build. This overrides
GO_VERSION in the buildroot go package.

Testing shows that go is installed from binary release, so we have now
checksums for both source and binary releases for amd64 and arm64.
Buildroot builtin go.hash does the same[3].

[1] https://storage.cloud.google.com/minikube-builds/logs/22099/0a27ed4/iso_build.txt
[2] https://pkg.go.dev/strings#SplitSeq
[3] https://github.com/buildroot/buildroot/blob/master/package/go/go.hash
@nirs
Copy link
Contributor Author

nirs commented Dec 11, 2025

/ok-to-build-iso

@minikube-bot
Copy link
Collaborator

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

@nirs
Copy link
Contributor Author

nirs commented Dec 11, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Dec 11, 2025
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 22101 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 39.0s    │ 39.1s                  │
│ enable ingress │ 15.8s    │ 15.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 38.5s 40.4s 37.8s 39.9s 38.5s
Times for minikube (PR 22101) start: 38.9s 39.7s 38.0s 40.3s 38.8s

Times for minikube ingress: 15.7s 15.8s 15.8s 15.8s 15.7s
Times for minikube (PR 22101) ingress: 14.8s 16.3s 15.8s 15.7s 15.8s

docker driver with docker runtime

┌───────────────────┬──────────┬────────────────────────┐
│      COMMAND      │ MINIKUBE │ MINIKUBE  ( PR 22101 ) │
├───────────────────┼──────────┼────────────────────────┤
│ minikube start    │ 22.2s    │ 22.8s                  │
│ ⚠️  enable ingress │ 10.6s    │ 22.6s ⚠️                │
└───────────────────┴──────────┴────────────────────────┘

Times for minikube ingress: 10.6s 10.6s 10.6s 10.6s 10.6s
Times for minikube (PR 22101) ingress: 10.6s 40.6s 10.6s 40.6s 10.6s

Times for minikube start: 23.0s 23.5s 23.8s 19.8s 20.7s
Times for minikube (PR 22101) start: 20.4s 22.9s 23.3s 22.8s 24.7s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 22101 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 20.7s    │ 20.9s                  │
│ enable ingress │ 20.5s    │ 20.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube ingress: 20.1s 20.1s 21.1s 21.1s 20.1s
Times for minikube (PR 22101) ingress: 21.1s 20.2s 20.1s 21.1s 21.1s

Times for minikube start: 18.4s 22.2s 23.8s 18.5s 20.8s
Times for minikube (PR 22101) start: 20.3s 22.1s 21.7s 18.4s 22.0s

@k8s-ci-robot
Copy link
Contributor

@nirs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
integration-vfkit-docker-macos-arm64 e65f928 link false /test integration-vfkit-docker-macos-arm64
integration-docker-docker-linux-x86-64 e65f928 link true /test integration-docker-docker-linux-x86-64
integration-docker-docker-linux-arm64 e65f928 link true /test integration-docker-docker-linux-arm64
integration-docker-crio-linux-x86-64 e65f928 link true /test integration-docker-crio-linux-x86-64
integration-kvm-crio-linux-x86-64 e65f928 link true /test integration-kvm-crio-linux-x86-64

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@medyagh
Copy link
Member

medyagh commented Dec 12, 2025

/lgtm

@medyagh medyagh merged commit 670b947 into kubernetes:master Dec 12, 2025
28 of 50 checks passed
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 12, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 12, 2025
@minikube-pr-bot
Copy link

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment Test Name Flake Rate
Docker_Linux (1 failed) TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/ConfigCmd(gopogh) Unknown

Besides the following environments also have failed tests:

  • Docker_Linux_crio: 27 failed (gopogh)

  • KVM_Linux_crio: 13 failed (gopogh)

  • Docker_Linux_crio_arm64: 40 failed (gopogh)

  • Docker_Linux_containerd_arm64: 34 failed (gopogh)

To see the flake rates of all tests by environment, click here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants