Podman exporter cannot find crun

[poupryc]

Describe the bug
I'm using a Podman instance installed with Homebrew, which changes the default paths for binaries such as netavark or crun.

When launched in the terminal, prometheus-podman-exporter runs fine:

poupryc@home:~$ prometheus-podman-exporter 
time=2025-03-18T10:05:19.922Z level=INFO source=exporter.go:67 msg="starting podman-prometheus-exporter" version="(version=1.15.0, branch=, revision=1)"
time=2025-03-18T10:05:19.922Z level=INFO source=exporter.go:68 msg=metrics enhanced=false
time=2025-03-18T10:05:19.922Z level=INFO source=handler.go:92 msg="enabled collectors"
time=2025-03-18T10:05:19.922Z level=INFO source=handler.go:103 msg=collector name=container
time=2025-03-18T10:05:19.926Z level=INFO source=container.go:239 msg="starting container size cache ticker" duration=3600
time=2025-03-18T10:05:19.926Z level=INFO source=container.go:240 msg="update container size cache"
time=2025-03-18T10:05:19.926Z level=INFO source=events.go:16 msg="starting podman event streamer"
time=2025-03-18T10:05:19.929Z level=INFO source=exporter.go:95 msg="Listening on" address=:9882
time=2025-03-18T10:05:19.929Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9882
time=2025-03-18T10:05:19.929Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9882

However, when used with systemd and an User and Group specified (for a rootless Podman setup), it fails:

Mar 18 10:59:46 poupryc-ThinkPad-P14s-Gen-4 systemd[1]: Started prometheus-podman-exporter-test.service - Prometheus exporter for podman (v4) machine.
Mar 18 10:59:46 poupryc-ThinkPad-P14s-Gen-4 prometheus-podman-exporter[22037]: time=2025-03-18T09:59:46.399Z level=INFO source=exporter.go:67 msg="starting podman-prometheus-exporter" version="(version=1.15.0, branch=, revision=1)"
Mar 18 10:59:46 poupryc-ThinkPad-P14s-Gen-4 prometheus-podman-exporter[22037]: time=2025-03-18T09:59:46.400Z level=INFO source=exporter.go:68 msg=metrics enhanced=false
Mar 18 10:59:46 poupryc-ThinkPad-P14s-Gen-4 prometheus-podman-exporter[22037]: time=2025-03-18T09:59:46.400Z level=INFO source=handler.go:92 msg="enabled collectors"
Mar 18 10:59:46 poupryc-ThinkPad-P14s-Gen-4 prometheus-podman-exporter[22037]: time=2025-03-18T09:59:46.400Z level=INFO source=handler.go:103 msg=collector name=container
Mar 18 10:59:46 poupryc-ThinkPad-P14s-Gen-4 prometheus-podman-exporter[22037]: 2025/03/18 10:59:46 default OCI runtime "crun" not found: invalid argument
Mar 18 10:59:46 poupryc-ThinkPad-P14s-Gen-4 systemd[1]: prometheus-podman-exporter-test.service: Main process exited, code=exited, status=1/FAILURE

To Reproduce
Steps to reproduce the behavior:

Install podman using brew using brew install podman

Verify the crun path using podman info:

  ociRuntime:
    name: crun
    package: Unknown
    path: /home/linuxbrew/.linuxbrew/bin/crun
    version: |-
      crun version 1.20

Reproduce the error by running:

$ sudo runuser -u poupryc -g poupryc prometheus-podman-exporter
time=2025-03-18T10:14:33.124Z level=INFO source=exporter.go:67 msg="starting podman-prometheus-exporter" version="(version=1.15.0, branch=, revision=1)"
time=2025-03-18T10:14:33.124Z level=INFO source=exporter.go:68 msg=metrics enhanced=false
time=2025-03-18T10:14:33.124Z level=INFO source=handler.go:92 msg="enabled collectors"
time=2025-03-18T10:14:33.124Z level=INFO source=handler.go:103 msg=collector name=container
2025/03/18 11:14:33 default OCI runtime "crun" not found: invalid argument

Providing the PATH for the Homebrew bin directory crashes prometheus-podman-exporter:

$ sudo runuser -l poupryc -g poupryc -c "PATH=/home/linuxbrew/.linuxbrew/bin:/usr/bin prometheus-podman-exporter"
time=2025-03-18T10:19:43.522Z level=INFO source=exporter.go:67 msg="starting podman-prometheus-exporter" version="(version=1.15.0, branch=, revision=1)"
time=2025-03-18T10:19:43.522Z level=INFO source=exporter.go:68 msg=metrics enhanced=false
time=2025-03-18T10:19:43.522Z level=INFO source=handler.go:92 msg="enabled collectors"
time=2025-03-18T10:19:43.522Z level=INFO source=handler.go:103 msg=collector name=container
time=2025-03-18T10:19:43.524Z level=INFO source=container.go:239 msg="starting container size cache ticker" duration=3600
time=2025-03-18T10:19:43.524Z level=INFO source=container.go:240 msg="update container size cache"
time=2025-03-18T10:19:43.524Z level=INFO source=events.go:16 msg="starting podman event streamer"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x144784f]

goroutine 1 [running]:
github.com/containers/podman/v5/libpod.(*Container).rwSize(0xc0003292a0?)
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/containers/podman/v5/libpod/container_internal.go:104 +0x2f
github.com/containers/podman/v5/libpod.(*Container).RWSize(0x63?)
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/containers/podman/v5/libpod/container.go:1208 +0xd0
github.com/containers/podman/v5/pkg/ps.ListContainerBatch.func1(0xc0003f0000)
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/containers/podman/v5/pkg/ps/ps.go:227 +0xe77
github.com/containers/podman/v5/libpod.(*Container).Batch(0xc0002c7d60, 0xc00063ce78)
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/containers/podman/v5/libpod/container_api.go:935 +0x136
github.com/containers/podman/v5/pkg/ps.ListContainerBatch(_, _, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...})
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/containers/podman/v5/pkg/ps/ps.go:158 +0x30c
github.com/containers/podman/v5/pkg/ps.GetContainerLists(0xc000103c00, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, 0x0, ...})
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/containers/podman/v5/pkg/ps/ps.go:77 +0x509
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).ContainerList(0x15b99ec?, {0x15b99ec?, 0xc1ee700fdf489122?}, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, ...})
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/containers/podman/v5/pkg/domain/infra/abi/containers.go:1058 +0x58
github.com/containers/prometheus-podman-exporter/pdcs.updateContainerSize()
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/pdcs/container.go:213 +0xf0
github.com/containers/prometheus-podman-exporter/pdcs.StartCacheSizeTicker(0xc0001fe2d0, 0xe10)
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/pdcs/container.go:242 +0xe5
github.com/containers/prometheus-podman-exporter/exporter.Start(0x2a00660, {0x0?, 0x0?, 0x0?})
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/exporter/exporter.go:93 +0x474
github.com/containers/prometheus-podman-exporter/cmd.run(0x2a00660?, {0x2b235a0?, 0x4?, 0x1aeee57?})
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/cmd/root.go:53 +0x1c
github.com/spf13/cobra.(*Command).execute(0x2a00660, {0xc000040090, 0x0, 0x0})
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/spf13/cobra/command.go:1019 +0xa9b
github.com/spf13/cobra.(*Command).ExecuteC(0x2a00660)
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/spf13/cobra/command.go:1148 +0x40c
github.com/spf13/cobra.(*Command).Execute(...)
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/vendor/github.com/spf13/cobra/command.go:1071
github.com/containers/prometheus-podman-exporter/cmd.Execute()
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/cmd/root.go:61 +0x1e
main.main()
	/tmp/tmp.iGTmdpxBZf/prometheus-podman-exporter-1.15.0/main.go:8 +0xf

Expected behavior
prometheus-podman-exporter should be able to find the crun runtime based on the libpod.conf runtimes key and should be able to launch inside systemd with an active User and Group configuration.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu 24.04.2 LTS
• Podman Version

Client:        Podman Engine
Version:       5.4.1
API Version:   5.4.1
Go Version:    go1.24.1
Built:         Tue Mar 11 18:22:13 2025
Build Origin:  brew
OS/Arch:       linux/amd64

Additional context

• Brew 4.4.24

[navidys]

Hi @poupryc

I will look into it.

How did you install the exporter ?
Have u also tried the container image to see if it's working ?

Thanks

[poupryc]

Hello @navidys, thank you for your assistance!

• The exporter was compiled locally.
• The image works, but for several reasons, I need to use the non-containerized version of the exporter.

Have a great weekend

[navidys]

Hi @poupryc
Have you tried to set environment variable 'CONTAINERS_CONF' which will point to your podman configuration.

export CONTAINERS_CONF=path
prometheus-podman-exporter -a

[karaolidis]

Encountered a similar issue when trying to package the application for NixOS:

{ pkgs, ... }:
# AUTO-UPDATE: nix-update --flake prometheus-podman-exporter
pkgs.buildGoModule rec {
  pname = "prometheus-podman-exporter";
  version = "1.15.0";

  src = pkgs.fetchFromGitHub {
    owner = "containers";
    repo = "prometheus-podman-exporter";
    rev = "v${version}";
    hash = "sha256-eXuLiJx0WsPlPAN5ZwQIp89thXiNS6AGE9p3aqjD+K8=";
  };

  vendorHash = null;

  nativeBuildInputs = with pkgs; [
    makeWrapper
    pkg-config
  ];

  buildInputs = with pkgs; [
    btrfs-progs
    gpgme
    systemd
  ];

  GOFLAGS = [ "-tags='systemd'" ];

  ldflags = [
    "-X github.com/containers/prometheus-podman-exporter/cmd.buildVersion=${version}"
    "-X github.com/containers/prometheus-podman-exporter/cmd.buildRevision=${builtins.head (pkgs.lib.strings.splitString "." version)}"
    "-X github.com/containers/prometheus-podman-exporter/cmd.buildBranch=HEAD"
  ];

  # Don't run tests because they require a running podman daemon
  doCheck = false;
}

This leads to:

> nix build .\#prometheus-podman-exporter && ./result/bin/prometheus-podman-exporter -d
warning: Git tree '/home/nikara/git/nix' is dirty
time=2025-04-10T14:32:54.439Z level=INFO source=exporter.go:67 msg="starting podman-prometheus-exporter" version="(version=1.15.0, branch=HEAD, revision=1)"
time=2025-04-10T14:32:54.439Z level=INFO source=exporter.go:68 msg=metrics enhanced=false
time=2025-04-10T14:32:54.439Z level=INFO source=handler.go:92 msg="enabled collectors"
time=2025-04-10T14:32:54.439Z level=INFO source=handler.go:103 msg=collector name=container
2025/04/10 15:32:54 could not find a working conmon binary (configured options: [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon]: invalid argument)

[karaolidis]

Building the remote version of the binary and wrapping it with an env var seemed to do the trick for me:

{ pkgs, ... }:
# AUTO-UPDATE: nix-update --flake prometheus-podman-exporter
pkgs.buildGoModule rec {
  pname = "prometheus-podman-exporter";
  version = "1.15.0";

  src = pkgs.fetchFromGitHub {
    owner = "containers";
    repo = "prometheus-podman-exporter";
    rev = "v${version}";
    hash = "sha256-eXuLiJx0WsPlPAN5ZwQIp89thXiNS6AGE9p3aqjD+K8=";
  };

  vendorHash = null;

  nativeBuildInputs = with pkgs; [
    makeWrapper
    pkg-config
  ];

  buildInputs = with pkgs; [
    btrfs-progs
    gpgme
  ];

  tags = [
    "containers_image_openpgp"
    "remote"
  ];

  ldflags = [
    "-s"
    "-w"
    "-X github.com/containers/prometheus-podman-exporter/cmd.buildVersion=${version}"
    "-X github.com/containers/prometheus-podman-exporter/cmd.buildRevision=${builtins.head (pkgs.lib.strings.splitString "." version)}"
    "-X github.com/containers/prometheus-podman-exporter/cmd.buildBranch=HEAD"
  ];

  # Don't run tests because they require a running podman daemon
  doCheck = false;

  postInstall = ''
    wrapProgram $out/bin/prometheus-podman-exporter \
      --run "export CONTAINER_HOST=\"unix://\$(test -S \''${XDG_RUNTIME_DIR:-/run/user/\$(id -u)}/podman/podman.sock && echo \''${XDG_RUNTIME_DIR:-/run/user/\$(id -u)}/podman/podman.sock || echo /run/podman/podman.sock)\""
  '';
}

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.