"stat /pause: operation not supported" when creating a cluser with rootless podman driver and a backing filesystem without POSIX acl

[chaserhkj]

What happened:

I could not create a cluster using kind's rootless podman driver, the command simply fails with the healthcheck on apiserver after 4mins of timeout.

I looked into the journals of the control plane container and it seems that it failed to start the nested containers for the control plane services with a failure message that reads "stat /pause: operation not supported"

Full logs attached below

logs.zip

What you expected to happen:

Cluster creation succeeds

How to reproduce it (as minimally and precisely as possible):

$ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --retain

Anything else we need to know?:

It seems that this is strictly related to rootless podman on my system, since rootful cluster creation is just fine:

$ sudo env KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --retain
[sudo] password for hkj:
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.34.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

So I feel this might have something to do with my system setup, and some unwanted interactions with seccomp and other security sandbox features.

Environment:

• kind version: (use kind version): kind v0.30.0 go1.24.6 linux/amd64
• Runtime info: (use docker info, podman info or nerdctl info):

host:
  arch: amd64
  buildahVersion: 1.41.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 90.5
    systemPercent: 1.35
    userPercent: 8.14
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2011
  hostname: hkj-desktop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 131073
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 131073
  kernel: 6.16.4-zen1-1-zen
  linkmode: dynamic
  logDriver: journald
  memFree: 14882959360
  memTotal: 67327676416
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1
    path: /usr/lib/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_08_05.309eefd-1
    version: |
      pasta 2025_08_05.309eefd
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.3-1
    version: |-
      slirp4netns version 1.3.3
      commit: 944fa94090e1fd1312232cbc0e6b43585553d824
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.5.6
  swapFree: 0
  swapTotal: 0
  uptime: 734h 20m 48.00s (Approximately 30.58 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/hkj/.config/containers/storage.conf
  containerStore:
    number: 12
    paused: 0
    running: 5
    stopped: 7
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/hkj/.local/share/containers/storage
  graphRootAllocated: 1624816222208
  graphRootUsed: 423147208704
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 432
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/hkj/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.0
  Built: 1755376297
  BuiltTime: Sat Aug 16 16:31:37 2025
  GitCommit: da671ef6cfa3fc9ac6225c18f1dd0a70a951e43f
  GoVersion: go1.25.0
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.0

• OS (e.g. from /etc/os-release):

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

• Kubernetes version: (use kubectl version):

Client Version: v1.33.4
Kustomize Version: v5.6.0
The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Any proxies or other special environment settings?: N/A

[BenTheElder]

Have you checked:
https://kind.sigs.k8s.io/docs/user/known-issues/
https://kind.sigs.k8s.io/docs/user/rootless/

[chaserhkj]

I did set Delegate=yes on my systemd user daemon:

$ systemctl cat user@1000.service
# /usr/lib/systemd/system/user@.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=User Manager for UID %i
Documentation=man:user@.service(5)
BindsTo=user-runtime-dir@%i.service
After=systemd-logind.service user-runtime-dir@%i.service dbus.service systemd-oomd.service
IgnoreOnIsolate=yes

[Service]
User=%i
PAMName=systemd-user
Type=notify-reload
ExecStart=/usr/lib/systemd/systemd --user
Slice=user-%i.slice
# Reexecute the manager on service reload, instead of reloading.
# This provides a synchronous method for restarting all user manager
# instances after upgrade.
ReloadSignal=RTMIN+25
KillMode=mixed
Delegate=pids memory cpu
DelegateSubgroup=init.scope
TasksMax=infinity
TimeoutStopSec=120s
KeyringMode=inherit
OOMScoreAdjust=100
MemoryPressureWatch=skip

# /usr/lib/systemd/system/user@.service.d/10-login-barrier.conf
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
# Make sure user instances are started after logins are allowed. However this
# is not desirable for user@0.service since root should be able to log in
# earlier during the boot process especially if something goes wrong.
After=systemd-user-sessions.service

# /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes

And all iptables kmod are loaded as well:

$ lsmod | grep 'ip.*table'
ip6table_raw           12288  0
iptable_raw            12288  0
ip6table_mangle        12288  2
ip6table_filter        12288  3
iptable_filter         12288  3
iptable_mangle         12288  2
ip6table_nat           12288  3
ip6_tables             36864  4 ip6table_filter,ip6table_raw,ip6table_nat,ip6table_mangle
iptable_nat            12288  3
nf_nat                 69632  6 ip6table_nat,xt_nat,nft_masq,nft_chain_nat,iptable_nat,xt_MASQUERADE
ip_tables              36864  4 iptable_filter,iptable_raw,iptable_nat,iptable_mangle
x_tables               65536  27 ip6table_filter,xt_conntrack,xt_statistic,ip6table_raw,iptable_filter,ip6table_nat,xt_multiport,xt_tcpudp,xt_addrtype,xt_CHECKSUM,xt_nat,ip6t_rt,xt_comment,xt_set,ip6_tables,ipt_REJECT,xt_nfacct,iptable_raw,ip_tables,iptable_nat,xt_limit,xt_hl,ip6table_mangle,xt_MASQUERADE,ip6t_REJECT,iptable_mangle,xt_mark

Strangely though none of the systemd-run command on the rootless doc page works on my system:

$ systemd-run --scope --user kind create cluster
Running as unit: run-p2043042-i27198183.scope; invocation ID: 7988c015ef494f1cbe1a0f9b21613bfb
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
$ systemd-run --scope --user -p "Delegate=yes" kind create cluster
Running as unit: run-p2043296-i27198437.scope; invocation ID: 8a049036c755435f9d657929b1466b56
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

I did not find any relevant entries from the known issues page.

[BenTheElder]

/area provider/podman
See also #4022 for active discussion about the rootless docs and podman in particular

Unfortunately I cannot dedicate a lot of time to podman issues myself, I think this also goes for @aojea and @stmcginnis.

It's hard to justify when the other runtimes work with kind ~reliably and are sufficient for developing Kubernetes and my dayjob is primarily Kubernetes itself (which uses docker for development) while I personally use docker + limactl at home, but I can review proposed fixes and provide some pointers.

At this point I lean towards simply not recommending it, because of the frequency of new versions breaking something.
Granted, kind is doing things most normal containers would not, by necessity.

https://github.com/kubernetes-sigs/kind/issues?q=is%3Aissue%20state%3Aopen%20label%3Aarea%2Fprovider%2Fpodman
https://github.com/kubernetes-sigs/kind/issues?q=is%3Aissue%20state%3Aopen%20label%3Aarea%2Fprovider%2Fdocker
https://github.com/kubernetes-sigs/kind/issues?q=is%3Aissue%20state%3Aopen%20label%3Aarea%2Fprovider%2Fnerdctl

cc @jberkus

Naively: you might double check if systemd was reloaded with the daemon config changes.

EDIT: It looks like mount permission is blocked to the container processes. That won't be from cgroup delegation.

[AkihiroSuda]

Backing Filesystem: zfs

Looks somewhat unusual.
Worth trying ext4 or xfs

[chaserhkj]

@BenTheElder Thanks for the insight. I totally understand that podman driver might not be of higher priority for the team. Let me see if I could figure out anything here myself and propose a fix.

@AkihiroSuda A bit tricky to switch fs on my setup, will give it a try later.

[chaserhkj]

Looks like it is indeed a compatibility problem with zfs, I successfully created a ext4 graphroot and it worked.

Steps I took:

1. Create a guest user on my workstation (my main user has long running rootless podman services that I do not want to interrupt)
2. Create a zvol device (virtual block device backed by zpool storage from zfs)
3. mkfs.ext4 on zvol and mount it to /home/guest/.local/share/containers, then do proper chown for it
4. Set /etc/subuid /etc/subgid for guest user and loginctl enable-linger guest (required for rootless podman to work, these are already done for my main user prior to this post)
5. sudo -iu guest and now podman info gives:

host:
  arch: amd64
  buildahVersion: 1.41.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 90.51
    systemPercent: 1.35
    userPercent: 8.14
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2046
  hostname: hkj-desktop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 200000
      size: 231073
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 200000
      size: 231073
  kernel: 6.16.4-zen1-1-zen
  linkmode: dynamic
  logDriver: journald
  memFree: 9914908672
  memTotal: 67327676416
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1
    path: /usr/lib/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_08_05.309eefd-1
    version: |
      pasta 2025_08_05.309eefd
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.3-1
    version: |-
      slirp4netns version 1.3.3
      commit: 944fa94090e1fd1312232cbc0e6b43585553d824
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.5.6
  swapFree: 0
  swapTotal: 0
  uptime: 738h 4m 13.00s (Approximately 30.75 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/guest/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/guest/.local/share/containers/storage
  graphRootAllocated: 10464022528
  graphRootUsed: 2343112704
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/guest/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.0
  Built: 1755376297
  BuiltTime: Sat Aug 16 16:31:37 2025
  GitCommit: da671ef6cfa3fc9ac6225c18f1dd0a70a951e43f
  GoVersion: go1.25.0
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.0

Backing Filesystem is now extfs

Then kind just works with podman driver:

[guest@hkj-desktop containers]$ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --retain
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.34.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

I am not exactly sure why kind does not work on zfs backed graphRoot, though. Since in the past I have run nested container on podman with zfs as backing FS following guidelines here. It didn't create any problem for me. I'll try to dig zfs documentations for any clues there.

[chaserhkj]

Changed title to reflect latest found scopes of this issue.

[chaserhkj]

After some digging I found an old issue of containers/podman#11213 that seems to relate to this. And indeed my home zfs dataset has acltype=off. After setting it to acltype=posix and re-pulling the node image (to recreate the inodes with acl data), kind seems to work right now on top of my zfs paths.

So it seems that the error is thrown when the underlying file system does not have acl support but fuse-overlayfs tries to expose acl on top of an incapable FS. (This also explains why it works in rootful mode since kernel-level overlayfs is used in that case and no fuse-overlayfs is involved) According to the issue thread mentioned, upstream fuse-overlayfs implemented -o noacl to disable acl exposure and avoid this error. And it seems that version differences and/or configuration differences in fedora's nested podman image and kind's node image cause different default behaviors of fuse-overlayfs on two sides, resulting in kind unable to run nested images but podman image able to.

I think this should be an issue handled by containerd since they are the party actually calling fuse-overlayfs in kind nodes and it's up to their decision if they should call it with noacl on FS without an underlying acl.

For now I would suggest we add to the rootless documentation page (and maybe known issues page as well) that the underlying FS must support ACL as a host requirement of rootless kind. Maybe with some tips particularly with ZFS, too.

[jberkus]

@giuseppe @mheon is Kind-on-Podman currently a priority for anyone in Podman? I don't think it is, but I'm not the most up to date.

[mheon]

We're not testing Kind in our CI, but even if we were, I don't think we would have caught this - this seems very ZFS specific, and we're not doing any validation or testing of Podman on ZFS upstream.

[BenTheElder]

We have some limited podman CI these days but it's impractical to include HEAD of other projects, for that we're focused on Kubernetes (again for SIG Testing which is sponsoring, this is the main goal).

In any case this particular issue turns out to be related to containers on ZFS.

ZFS might well have had issues on any container runtime, it's not well tested elsewhere either AFAIK. I recommend something like ext4 in really wide use for backing containers (can mount other volumes for persistent storage).

Comments about lack of time and backlog for podman stand though ... https://github.com/kubernetes-sigs/kind/issues?q=state%3Aopen%20label%3A%22area%2Fprovider%2Fpodman%22

EG #4006

I don't expect podman maintainers to pay attention to any of this.
We're not getting much help from anyone though, and the pace of new major versions with breaking changes has been distracting by comparison (e.g. Netavark).

TBF I think podman also suffers from a higher rate of users that are more willing to tinker with system configurations though, which isn't inherent to podman, but is harder to support ...

[chaserhkj]

I would the say the problem is not with ZFS per se, but with any backing FS that lacks a POSIX-style ACL. It is just that most modern Linux FS support this by default, ZFS not having it by default is just a strange legacy of the past. I'll change the title to reflect this more precisely.

And about the podman discussion, just curious, how practical it is to separate the provider logic as a plugin and just put podman provider support as a side project of kind with community support?