Not able to run the exporter in rootless container environment #416
Description
We have multiple containers running in a host as rootless containers.
Below are the environment details:
systemd version: systemd 239 (239-82.e18_10.5)
kernel version: 4.18.0-553.71.1.e18_10. x86.64
OS: RHEL-8
$ cat /etc/systemd/system/[email protected]/delegate.conf
[Service]
Delegate=cou cpuset io memory pids
$ uname -r
4.18.0-553.71.1.e18_10. x86.64
$ systemctl --version
systemd 239 (239-82.e18_10.5)
$ podman version
Client: Podman Engine
Version: 4.9.4-rhel
API Version: 4.9.4-rhel
Go Version: g01.23.9 (Red Hat 1.23.9-1.module+el8.10.0+23162+9223a61a)
Built: Wed Jun 25 12:17:23
OS/Arch: linux/amd64
$ podman info |grep -A5 cgroupControllers
cgroupControllers:
- memory
- pids
cgroupManager: cgroupfs
cgroupVersion: v2
When trying to run the exporter binary , facing a panic issue
$ ./prometheus-podman-exporter
ts=2025-09-17T13:00:39.205Z caller=exporter.go:68 level=info msg="Starting podman-prometheus-exporter" version="(version=1.10.0, branch=, revision=1)"
ts=2025-09-17T13:00:39.205Z caller=exporter.go:69 level=info msg=metrics enhanced=false
ts=2025-09-17T13:00:39.205Z caller=handler.go:94 level=info msg="enabled collectors"
ts=2025-09-17T13:00:39.205Z caller=handler.go:105 level=info collector=container
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x12f43ef]
goroutine 1 [running]:
github.com/containers/podman/v4/libpod.(*Container).rwSize(0x165b12c?)
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/containers/podman/v4/libpod/container_internal.go:100 +0x2f
github.com/containers/podman/v4/libpod.(*Container).RWSize(0x0?)
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/containers/podman/v4/libpod/container.go:1168 +0xbb
github.com/containers/podman/v4/pkg/ps.ListContainerBatch.func1(0xc000159980)
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/containers/podman/v4/pkg/ps/ps.go:227 +0xe48
github.com/containers/podman/v4/libpod.(*Container).Batch(0xc000126500, 0xc0004e0e68)
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/containers/podman/v4/libpod/container_api.go:892 +0x124
github.com/containers/podman/v4/pkg/ps.ListContainerBatch(_, _, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...})
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/containers/podman/v4/pkg/ps/ps.go:154 +0x2f7
github.com/containers/podman/v4/pkg/ps.GetContainerLists(0xc0001485a0, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, 0x0, ...})
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/containers/podman/v4/pkg/ps/ps.go:78 +0x5a5
github.com/containers/podman/v4/pkg/domain/infra/abi.(*ContainerEngine).ContainerList(0x0?, {0x0?, 0x0?}, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, ...})
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/containers/podman/v4/pkg/domain/infra/abi/containers.go:1042 +0x58
github.com/containers/prometheus-podman-exporter/pdcs.updateContainerSize()
/podman_exporter/prometheus-podman-exporter-1.10.0/pdcs/container.go:180 +0xec
github.com/containers/prometheus-podman-exporter/pdcs.StartCacheSizeTicker({0x1c5cf40, 0xc0000ef940}, 0xe10)
/podman_exporter/prometheus-podman-exporter-1.10.0/pdcs/container.go:208 +0x1d1
github.com/containers/prometheus-podman-exporter/exporter.Start(0x0?, {0x0?, 0x0?, 0x0?})
/podman_exporter/prometheus-podman-exporter-1.10.0/exporter/exporter.go:94 +0x5b6
github.com/containers/prometheus-podman-exporter/cmd.run(0x2a01e00?, {0x2afb5c0?, 0x4?, 0x19a973a?})
/podman_exporter/prometheus-podman-exporter-1.10.0/cmd/root.go:53 +0x1c
github.com/spf13/cobra.(*Command).execute(0x2a01e00, {0xc00003e230, 0x0, 0x0})
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/spf13/cobra/command.go:987 +0xb1b
github.com/spf13/cobra.(*Command).ExecuteC(0x2a01e00)
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/spf13/cobra/command.go:1115 +0x44f
github.com/spf13/cobra.(*Command).Execute(...)
/podman_exporter/prometheus-podman-exporter-1.10.0/vendor/github.com/spf13/cobra/command.go:1039
github.com/containers/prometheus-podman-exporter/cmd.Execute()
/podman_exporter/prometheus-podman-exporter-1.10.0/cmd/root.go:61 +0x1e
main.main()
/podman_exporter/prometheus-podman-exporter-1.10.0/main.go:8 +0xf
There are some articles out there with the podman version 4.9.4 and RHEL-8 we cannot run the exporter as rootless,
I'm able to run the exporter binary as a root though without the panic issue because all the required controllers are present when run as root.
Any recommendation / fixes to run the binary as rootless to collect cpu, io, memory, pids details.
I've deployed exporter v1.10.0 since i'm on podman version 4.9.4 and the binary is built from source.
if we have any pre-built binary already please suggest reference for the same
Tried the below solutions, but no luck:
- https://rootlesscontaine.rs/getting-started/common/cgroup2/
- https://unix.stackexchange.com/questions/624428/cgroups-v2-cgroup-controllers-not-delegated-to-non-privileged-users-on-centos-s
Any possibilities that we can run the exporter as root but collect all the different users container metrics without needing to run as rootless? Its not the same ask as [issues/400], but atleast i want to see for one pod atleast