Podman container with macvlan fails to restart via systemd — “Address already in use” error

[VergilGao]

When restarting a Podman container managed by systemd and using a macvlan network with DHCP, the container fails to start with the following error:

Error: netavark: set macvlan up: Netlink error: Address already in use (os error 98)

This happens consistently when using systemctl restart on the service unit. The container starts fine initially, but fails on restart — even after removing it with podman rm -f. Only a full system reboot clears the issue.

Network Configuration (macvlan.network):

[Network]
DisableDNS=true
Driver=macvlan
IPAMDriver=dhcp
PodmanArgs=--interface-name eth3

Container Configuration (mqtt-broker.container):

[Unit]
Description=mqtt broker server
After=network-online.target
Wants=network-online.target
Before=shutdown.target

[Container]
HostName=mqtt-broker
AutoUpdate=registry
Image=docker.io/eclipse-mosquitto:latest
Network=macvlan.network
#PodmanArgs=--ip 192.168.16.6
PodmanArgs=--mac-address 60:8C:E0:38:D7:8A
Volume=/srv/docker/root/volumes/mqtt/data:/mosquitto/data
Volume=/srv/docker/root/volumes/mqtt/config:/mosquitto/config
Volume=/srv/docker/root/volumes/mqtt/log:/mosquitto/log

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

Steps to Reproduce:

Start the container via systemd:

sudo systemctl start mqtt-broker.service

Container starts successfully

IP is correctly assigned via DHCP

Restart the container:

sudo systemctl restart mqtt-broker.service

Container fails to start

Error: Netlink error: Address already in use

Inspect DHCP lease:

File /run/podman/nv-proxy.lease still contains the previous lease

Deleting this file has no effect

Killing nv-proxy also has no effect — it auto-restarts

The lease file is not regenerated after restart

Reboot the host:

After reboot, container starts successfully again

Additional Observation:

The container has AutoUpdate=registry enabled in the systemd unit

Automatic updates work fine — the container is replaced and restarted without issue

Manual restarts via systemctl restart consistently trigger the MAC address conflict

[Luap99]

Mhh, I assume quite a few folks are using macvlan + dhcp and this is the first time this comes up. Can you share the podman info output, what kernel is this running on?

I find it weird that we get EADDRINUSE when bringing up the interface (ip link set up) basically. Does the kernel only check the macs when bringing up the interface and then error? But the old interface and namespace should removed on contianer stop already so by the time the new container is started it no longer exists.

Can you check /run/netns for the ns files? Is it possible that podman somehow leaks them.

[VergilGao]

Mhh, I assume quite a few folks are using macvlan + dhcp and this is the first time this comes up. Can you share the podman info output, what kernel is this running on?

I find it weird that we get EADDRINUSE when bringing up the interface (ip link set up) basically. Does the kernel only check the macs when bringing up the interface and then error? But the old interface and namespace should removed on contianer stop already so by the time the new container is started it no longer exists.

Can you check /run/netns for the ns files? Is it possible that podman somehow leaks them.

I'm currently at work, and due to some recent issues with my home network, I'm temporarily unable to connect to my home environment.

At the moment, I'm running Fedora 42 inside an LXC container installed on OpenWRT. Within that Fedora instance, I use Podman to manage my containers. OpenWRT is also providing DHCP services. Since OpenWRT doesn't support systemd for daemon management, I believe this nested setup is currently the most suitable solution for my needs.

I'll be able to share more detailed information after I finish work. If there's anything you need me to help debug, I'll do my best to assist.

I'm based in the UTC+8 time zone (Beijing Time), so there may be some delay in my responses due to the time difference. Apologies in advance for any inconvenience.

I'm committed to helping resolve this issue as best I can!

[Luap99]

At the moment, I'm running Fedora 42 inside an LXC container installed on OpenWRT. Within that Fedora instance, I use Podman to manage my containers. OpenWRT is also providing DHCP services. Since OpenWRT doesn't support systemd for daemon management, I believe this nested setup is currently the most suitable solution for my needs.

Ack, yeah that is certainly a more niche setup. My only logical guess would be that we leak the old netns/interface somehow so the mac stays allocated.

podman inspect --format {{.NetworkSettings.SandboxKey}} <name> should show the you the namespace path used for the container. When the container is stopped that one should be deleted, and the next start creates a new random one.

If you comment out PodmanArgs=--mac-address 60:8C:E0:38:D7:8A does it work? That would confirm the EADDRINUSE errors is indeed about using the same mac address.

[VergilGao]

At the moment, I'm running Fedora 42 inside an LXC container installed on OpenWRT. Within that Fedora instance, I use Podman to manage my containers. OpenWRT is also providing DHCP services. Since OpenWRT doesn't support systemd for daemon management, I believe this nested setup is currently the most suitable solution for my needs.

Ack, yeah that is certainly a more niche setup. My only logical guess would be that we leak the old netns/interface somehow so the mac stays allocated.

podman inspect --format {{.NetworkSettings.SandboxKey}} <name> should show the you the namespace path used for the container. When the container is stopped that one should be deleted, and the next start creates a new random one.

If you comment out PodmanArgs=--mac-address 60:8C:E0:38:D7:8A does it work? That would confirm the EADDRINUSE errors is indeed about using the same mac address.

OK

# meo @ cattery in ~ [17:49:19] C:1                                                                                                                                                         $ sudo podman inspect --format {{.NetworkSettings.SandboxKey}} systemd-mqtt-broker                                                                                                          /run/user/0/netns/netns-936f0445-54bc-08e6-a4ad-a323a1975a6c                                                                                                                                                                                                                                                                                                                            # meo @ cattery in ~ [17:49:34]                                                                                                                                                             $ sudo systemctl restart mqtt-broker.service                                                                                                                                                Job for mqtt-broker.service failed because the control process exited with error code.                                                                                                      See "systemctl status mqtt-broker.service" and "journalctl -xeu mqtt-broker.service" for details.

then i switch to root user:

root@cattery:/run/user/0/netns# ls -al /run/user/0/netns/
total 0
drwxr-xr-x 2 root   root   100 Aug 13 00:15 .
drwx------ 3 root   root    60 Aug 12 20:40 ..
-r--r--r-- 1 nobody nobody   0 Aug 13 00:15 netns-91a5451d-c7b6-d492-e26f-b0e2f3d787fb
-r--r--r-- 1 nobody nobody   0 Aug 12 20:40 netns-936f0445-54bc-08e6-a4ad-a323a1975a6c
-r--r--r-- 1 nobody nobody   0 Aug 12 20:40 netns-c727a028-5732-cd53-a16d-786fa09f0b19

when i comment out PodmanArgs=--mac-address 60:8C:E0:38:D7:8A
it works.
but i cant config the ip for this container in my dhcp server.

[Luap99]

That looks quite wrong, the netns files are excepted to be owned by root not nobody. And if they somehow end up being leaked that explain why it fails as the mac stays allocated.

Can you add GlobalArgs=--log-level=debug under the [Container] section to enable podman debug logs. And then please check the journal for any podman or netavark errors logged.

[VergilGao]

That looks quite wrong, the netns files are excepted to be owned by root not nobody. And if they somehow end up being leaked that explain why it fails as the mac stays allocated.

Can you add GlobalArgs=--log-level=debug under the [Container] section to enable podman debug logs. And then please check the journal for any podman or netavark errors logged.

i have a special lxc config， it may cause the tun device owned by nobody
i will give the config and debug log next day

[VergilGao]

That looks quite wrong, the netns files are excepted to be owned by root not nobody. And if they somehow end up being leaked that explain why it fails as the mac stays allocated.

Can you add GlobalArgs=--log-level=debug under the [Container] section to enable podman debug logs. And then please check the journal for any podman or netavark errors logged.

the lxc config:

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: -d ubuntu -r focal -a amd64
# Template script checksum (SHA-1): b27e730655b3208b5e2edcba69290c39970b4fd0
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0
lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file
security.nesting = true
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64

# Container specific configuration
lxc.idmap = u 0 100000 165536
lxc.idmap = g 0 100000 165536
lxc.rootfs.path = dir:/srv/lxc/containers/cattery/rootfs
lxc.uts.name = cattery

# Network configuration
lxc.net.0.type          = phys
lxc.net.0.link          = eth3
lxc.net.0.flags         = up
lxc.net.0.name          = eth3

the podman info

host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 92.17
    systemPercent: 1.84
    userPercent: 5.99
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "42"
  eventLogger: journald
  freeLocks: 2046
  hostname: cattery
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.6.73
  linkmode: dynamic
  logDriver: journald
  memFree: 927686656
  memTotal: 8082132992
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.2-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.15.2
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/sbin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.x86_64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 1406h 42m 38.00s (Approximately 58.58 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 511255052288
  graphRootUsed: 21862277120
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 7
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.5.2
  BuildOrigin: Fedora Project
  Built: 1750723200
  BuiltTime: Tue Jun 24 08:00:00 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.2

here is the debug log.
podman.log

[VergilGao]

Aug 16 10:17:37 cattery mqtt-broker[923]: time="2025-08-16T10:17:37+08:00" level=debug msg="Removing all exec sessions for container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac"
Aug 16 10:17:37 cattery /usr/bin/podman[933]: time="2025-08-16T10:17:37+08:00" level=debug msg="Checking if container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac should restart"
Aug 16 10:17:38 cattery mqtt-broker[923]: time="2025-08-16T10:17:38+08:00" level=debug msg="Container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac storage is already unmounted, skipping..."
Aug 16 10:17:38 cattery podman[923]: 2025-08-16 10:17:38.005566831 +0800 CST m=+0.108159569 container remove e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac (image=docker.io/library/eclipse-mosquitto:latest, name=systemd-mqtt-broker, PODMAN_SYSTEMD_UNIT=mqtt-broker.service, description=Eclipse Mosquitto MQTT Broker, io.containers.autoupdate=registry, maintainer=Roger Light <roger@atchoo.org>)
Aug 16 10:17:38 cattery mqtt-broker[923]: time="2025-08-16T10:17:38+08:00" level=debug msg="Failed to remove container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac: cleaning up container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac: removing container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac network: netavark (exit code 1): open container netns: open /run/user/0/netns/netns-8d6ab435-75cc-d321-4c72-3e7514ef1e3a: IO error: No such file or directory (os error 2)"
Aug 16 10:17:38 cattery mqtt-broker[923]: time="2025-08-16T10:17:38+08:00" level=debug msg="Multiple results for container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac - attempted multiple removals?"
Aug 16 10:17:38 cattery mqtt-broker[923]: Error: cleaning up container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac: removing container e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac network: netavark (exit code 1): open container netns: open /run/user/0/netns/netns-8d6ab435-75cc-d321-4c72-3e7514ef1e3a: IO error: No such file or directory (os error 2)
Aug 16 10:17:38 cattery mqtt-broker[923]: time="2025-08-16T10:17:38+08:00" level=debug msg="Shutting down engines"
Aug 16 10:17:38 cattery mqtt-broker[923]: time="2025-08-16T10:17:38+08:00" level=info msg="Received shutdown.Stop(), terminating!" PID=923
Aug 16 10:17:38 cattery /usr/bin/podman[933]: time="2025-08-16T10:17:38+08:00" level=debug msg="Called cleanup.PersistentPostRunE(/usr/bin/podman --root /var/lib/containers/storage --runroot /run/containers/storage --log-level debug --cgroup-manager systemd --tmpdir /run/libpod --network-config-dir  --network-backend netavark --volumepath /var/lib/containers/storage/volumes --db-backend sqlite --transient-store=false --runtime crun --storage-driver overlay --storage-opt overlay.imagestore=/usr/lib/containers/storage --storage-opt overlay.mountopt=nodev,metacopy=on --events-backend journald --syslog container cleanup --stopped-only --rm e1f9993541eaa81d2f9091bb6bd5aa0a19b6f822d105522168219be25cc7deac)"
Aug 16 10:17:38 cattery /usr/bin/podman[933]: time="2025-08-16T10:17:38+08:00" level=debug msg="Shutting down engines"
Aug 16 10:17:38 cattery /usr/bin/podman[933]: time="2025-08-16T10:17:38+08:00" level=info msg="Received shutdown.Stop(), terminating!" PID=933
Aug 16 10:17:38 cattery systemd[1]: mqtt-broker.service: Control process exited, code=exited, status=125/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStop= process belonging to unit mqtt-broker.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 125.

i found these log may tell us why.

maybe i should mount

lxc.mount.entry = /run/user/100000/netns run/user/0/netns none bind,optional,create=dir

into my lxc container.

i will try this tonight.

[VergilGao]

ok
i add these line in my mqtt-broker.container section [Service]

Environment=NETNS_MOUNT_DIR=/var/lib/netns
Environment=XDG_RUNTIME_DIR=/var/run

the problem fixed

the netns file shows in /run/netns/netns-0982007b-cddb-c937-8dbd-864ea2903958

$ ls -al /var/run/netns
total 0
drwxr-xr-x  2 root   root   100 Aug 19 21:41 .
drwxr-xr-x 27 root   root   720 Aug 19 21:41 ..
-r--r--r--  1 nobody nobody   0 Aug 19 21:41 netns-1b989618-967f-d67f-de8a-0cf70576fc17
-r--r--r--  1 nobody nobody   0 Aug 19 21:41 netns-8a5959f9-3c90-bf1e-9d0e-1c3c41311bda
-r--r--r--  1 nobody nobody   0 Aug 19 21:41 netns-bfc64bba-e075-449f-9ac6-7c5e4c4f8b16

the netns-xxxxx still belongs to nobody:nobody but they can remove now:

Aug 19 21:28:36 cattery mqtt-broker[852]: time="2025-08-19T21:28:36+08:00" level=debug msg="Tearing down network namespace at /run/netns/netns-0982007b-cddb-c937-8dbd-864ea2903958 for container b858b0babd48f56ab764337e461568b1ec03a4ba71966de92cbf532b925ae626"

update:
just add

Environment=XDG_RUNTIME_DIR=/var/run

it works fine.