Loading images with digest sometimes does not work

[ernado]

What happened:

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  6m27s                default-scheduler  Successfully assigned cilium/cilium-envoy-gxbkf to octo-worker
  Normal   Pulling    6m27s                kubelet            Pulling image "quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0"
  Normal   Pulled     6m8s                 kubelet            Successfully pulled image "quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0" in 1.689s (19.068s including waiting). Image size: 186478295 bytes.
  Normal   Pulled     71s (x24 over 6m7s)  kubelet            Container image "quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0" already present on machine
  Warning  Failed     60s (x26 over 6m8s)  kubelet            Error: failed to get image from containerd "sha256:5b9199b8f90ccc71ba939178d08bb25db735983514472342e1304b530a9b7d3c": image "docker.io/library/import-2025-10-07@sha256:cfa559fc37c7232c849bfd75a96b75a4ef74da46b4ecf2feb6480399a65f6670": not found

What you expected to happen:
Image found.

How to reproduce it (as minimally and precisely as possible):

Reproducer: https://github.com/ernado/kind-load-repro

kind create cluster --config kind.yml
docker pull quay.io/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667
docker pull quay.io/cilium/tetragon-operator:v1.5.0
docker pull quay.io/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805
docker pull quay.io/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d
docker pull quay.io/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953
docker pull quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0
docker pull quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac
docker pull quay.io/cilium/tetragon:v1.5.0
kind load docker-image quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0 quay.io/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667 quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac quay.io/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953 quay.io/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d quay.io/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805 quay.io/cilium/tetragon-operator:v1.5.0 quay.io/cilium/tetragon:v1.5.0
helm upgrade cilium cilium/cilium --install --values cilium.yml --namespace cilium --create-namespace --version 1.18.2

Then after some time:

kubectl get pods --all-namespaces --field-selector=status.phase!=Running,status.phase!=Succeeded | grep CreateContainerError

This is heisenbug, sometimes everything works.

Anything else we need to know?:
N/A

Environment:

• kind version: kind v0.30.0 go1.24.6 linux/amd64
• Runtime info:

Client: Docker Engine - Community
 Version:    28.5.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.29.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.39.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 23
  Running: 2
  Paused: 0
  Stopped: 21
 Images: 325
 Server Version: 28.5.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b98a3aace656320842a23f4a392a33f46af97866
 runc version: v1.3.0-0-g4ca628d1
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.0-33-generic
 Operating System: Ubuntu 24.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.7GiB
 Name: nexus
 ID: 5a4aa1ec-08d6-4acd-8cb1-eabcf8aacf57
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: ernado
 Experimental: false
 Insecure Registries:
  harbor.localhost
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release): Ubuntu 24.04.3 LTS
• Kubernetes version: (use kubectl version):

Client Version: v1.32.9
Kustomize Version: v5.5.0
Server Version: v1.34.

• Any proxies or other special environment settings?:
  N/A

[BenTheElder]

Is this different from the pinned issue #3795 ?

[BenTheElder]

Ah, loading a digest may not work, this is not something easily fixed in kind, this is behavior in docker (saving the image see #3975 for recent changes that might enable this but cause other problems) and containerd (importing the image)

https://github.com/ernado/kind-load-repro/blame/f4ff4f5ed3ea3d54d127bd7b8fd5a29761ea1f87/commands.sh#L12

[ernado]

I've tried trimming the digest, but still getting error:

  Normal   Pulled  3m37s (x1040 over 48m)  kubelet  Container image "quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac" already present on machine
  Warning  Failed  3m26s (x1041 over 48m)  kubelet  Error: failed to check if this is a checkpoint image: failed to get image from containerd "sha256:4456983f292e46335c26f66eb62bb708dc0a46f166c1466cf4c4892e6e81b563": image "docker.io/library/import-2025-10-07@sha256:0e1d8868652ecc11aa04fd856dc5d782efeb63d17b9366ae57e8b317f9a4afb7": not found

So there is condition when

1. Container image "quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac" already present on machine
2. Error: failed to check if this is a checkpoint image: failed to get image from containerd "sha256:4456983f292e46335c26f66eb62bb708dc0a46f166c1466cf4c4892e6e81b563": image "docker.io/library/import-2025-10-07@sha256:0e1d8868652ecc11aa04fd856dc5d782efeb63d17b9366ae57e8b317f9a4afb7": not found

[ernado]

I've forgot to mention that some images are loading properly and this is not #3795.

I have following condition:

 kubectl -n cilium get pods
NAME                               READY   STATUS                 RESTARTS        AGE
cilium-7bdff                       1/1     Running                1 (52m ago)     4h10m
cilium-envoy-f85rv                 1/1     Running                1 (52m ago)     4h10m
cilium-envoy-l4pvg                 1/1     Running                1 (52m ago)     4h10m
cilium-operator-79895854cb-2mnzh   0/1     CreateContainerError   1 (3h50m ago)   4h10m
cilium-t482d                       1/1     Running                1 (52m ago)     4h10m
hubble-relay-6479d9bd-x2lkh        0/1     CreateContainerError   0               4h10m
hubble-ui-576dcd986f-f5w5f         2/2     Running                2 (52m ago)     4h10m

Some pods are ok, some are having "failed go get image" error.
Sometimes it just works, and problematic images are not consistent.

[ernado]

I've re-tried and it looks like trimming digests works.

[BenTheElder]

I'm pretty sure it is related to #3795, are you certain it's not the digest pointing to the multi-arch manifest list?

[ernado]

In #3795:

• The kind load docker-image is failing
• It is failing every time
• Can be worked around with unticking "Use containerd for pulling and storing images"

In current issue:

• The docker pull is failing.
• It is failing sometimes, sometimes everything is ok
• I'm not using containerd for pulling and storing images and have no problem loading multiarch images without digests (I'm on ubuntu with default docker settings).

It looks like issues can be unrelated.

However, the image quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324 is actually multiarch.

[stmcginnis]

Confirmed quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324 is a multi-arch manifest:

crane manifest quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:4c2957e9f8fd094503d02c84d34f1ad04cb4cf455f14bcdce86e9f956a1b465e",
      "size": 901,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:f59ea3139d2312c90c83f282046712192c76033ebf684528dd7f84c87b9d173c",
      "size": 901,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ]
}

It is failing sometimes, sometimes everything is ok

Odd that it's not consistent. Do you have any setup helper scripts that might be setting DOCKER_DEFAULT_PLATFORM anywhere?

In the meantime, maybe you could update your script to pull the platform image directly:

 docker pull quay.io/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953
- docker pull quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0
+ docker pull quay.io/cilium/cilium-envoy@sha256:4c2957e9f8fd094503d02c84d34f1ad04cb4cf455f14bcdce86e9f956a1b465e
 docker pull quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac
 docker pull quay.io/cilium/tetragon:v1.5.0