[v1.0.1] Port forwarding fails under high concurrency load. (accept4: too many open files)

[Javier-VL]

Port forwarding fails under high concurrency load. Using ApacheBench with high concurrency causes the container to stop accepting connections.

Steps to reproduce:

• Start a rootless container with NGINX exposed on port 8081.
• Run ApacheBench from a remote host:
  ab -n 10000 -c 900 http://:8081/
• Observe that connections begin to fail:
  curl http://:8081/

curl: (7) Failed to connect to port 8081: Could not connect to server

Expected behavior:
Port forwarding should remain stable and accept connections even under high concurrency.
Actual behavior:
Connections are refused. Docker debug logs show:
port/builtin: accept: accept tcp4 0.0.0.0:8081: accept4: too many open files

...
Comparison with slirp4netns port driver
We tested the same setup using the slirp4netns port driver instead of builtin, by modifying the following line in dockerd-rootless.sh:
: "${DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER:=builtin}"

Changed to:
: "${DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER:=slirp4netns}"


Environment:

• slirp4netns version: 1.2.0-beta.0+dev
  Results:
• ApacheBench with ab -n 10000 -c 900 completes ~9764 requests.
• Occasionally hits:
  apr_socket_recv: Connection reset by peer (104)
• But ports remain open and responsive afterward.
• curl tests continue to work even after high load.

Conclusion:
While the issue with port/builtin may not be identical, switching to slirp4netns for port forwarding avoids the port closure behavior under high concurrency. Delegating port forwarding to slirp4netns appears to mitigate the "too many open files" error.

Reported a similar behavior at:,moby/moby#51248, while trying to hit a sporadic curl 56 error, im getting this errors instead

bash-5.1$ docker version
Client:
Version: 24.0.5
API version: 1.43
Go version: go1.20.7
Git commit: b74562d917
Built: Thu Oct 23 02:09:31 2025
OS/Arch: linux/amd64
Context: default
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
bash-5.1$ export DOCKER_HOST=unix:///run/user/500/docker.sock
bash-5.1$ docker version
Client:
Version: 24.0.5
API version: 1.43
Go version: go1.20.7
Git commit: b74562d917
Built: Thu Oct 23 02:09:31 2025
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 00e46f85f6e46bb4b02c33da253f901c473794e9
Built: Thu Oct 23 02:08:02 2025
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.7.20.m
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353.m
runc:
Version: 1.1.7+dev
GitCommit: v1.0.0-rc94-766-gb6109acd-dirty
docker-init:
Version: 0.19.0
GitCommit: b9f42a0-dirty
rootlesskit:
Version: 1.0.1
ApiVersion: 1.1.1
NetworkDriver: slirp4netns
PortDriver: slirp4netns
StateDir: /tmp/rootlesskit1616269797
slirp4netns:
Version: 1.2.0-beta.0+dev
GitCommit: unknown

[AkihiroSuda]

moby/moby#51248 (comment)
moby/moby#51248 (comment)

[AkihiroSuda]

Maybe you just have to adjust nofile
https://wiki.archlinux.org/title/Limits.conf

[Javier-VL]

I could increase my open files, currently i have a
ulimit -n 4096 for rootlesskit

However monitoring the fds, when being close to the set limit

I got

time="2025-10-27T12:57:25-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:8089: accept4: too many open files"
time="2025-10-27T12:57:25-06:00" level=debug msg="port/builtin: copyConnToChild: file file+net : fcntl: too many open files"
time="2025-10-27T12:57:25-06:00" level=debug msg="port/builtin: copyConnToChild: file file+net : fcntl: too many open files"
time="2025-10-27T12:57:25-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:8088: accept4: too many open files"

from apache bench, was getting, Connection reset by peer (104), until the port closes

sam@MEX8G7FJG3:~$ ab -n 60 -c 60 http://localhost:8089/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking ... (be patient)...apr_socket_recv: Connection reset by peer (104)
Total of 2 requests completed
sam@MEX8G7FJG3:~$ ab -n 60 -c 60 http://localhost:8089/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking ... (be patient)...apr_socket_recv: Connection reset by peer (104)
Total of 4 requests completed
sam@MEX8G7FJG3:~$ ab -n 60 -c 60 http://localhost:8089/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking ... (be patient)...apr_socket_recv: Connection refused (111)

The builtin driver is closing the port is this a bug? it appears is happening inside tcp.go

I guess I expected that would keep dropping connections and continue to accept instead of the port closing

[Javier-VL]

did a re-run. this time it closed all the ports

time="2025-10-27T14:56:45-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:27017: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:5672: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:15672: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:15671: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:8088: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:9061: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:9062: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:8089: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: accept: accept tcp4 0.0.0.0:5671: accept4: too many open files"
time="2025-10-27T14:56:46-06:00" level=debug msg="port/builtin: copyConnToChild: dial unix /tmp/rootlesskit1401936361/.bp.sock: socket: too many open files"

[AkihiroSuda]

Does that still happen with the latest version?
v1.0.1 is quite old.

[Javier-VL]

Yes, on a fedora wsl machine with Version: 2.3.5 of rootlesskit and a ulimit -n 4096
Stills happens
docker version
Client:
Version: 24.0.5
API version: 1.43
Go version: go1.20.6
Git commit: ced0996
Built: Fri Jul 21 20:34:32 2023
OS/Arch: linux/amd64
Context: default

Server: Docker Engine - Community
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.6
Git commit: a61e2b4
Built: Fri Jul 21 20:35:56 2023
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.7.20
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353
runc:
Version: 1.4.0-rc.1+dev
GitCommit: v1.4.0-rc.1-95-gfb01482d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
rootlesskit:
Version: 2.3.5
ApiVersion: 1.1.1
NetworkDriver: slirp4netns
PortDriver: builtin
StateDir: /tmp/rootlesskit2927113150
slirp4netns:
Version: 1.2.0
GitCommit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383

sam@MEX8G7FJG3:~/inbound$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4d9b478707d9 nginx:latest "/docker-entrypoint.…" About an hour ago Up 7 minutes 0.0.0.0:9999->80/tcp, :::9999->80/tcp nginx-v2-test

using apache bench

sam@MEX8G7FJG3:~/inbound$ ab -n 4096 -c 4096 http://localhost:9999/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
apr_socket_recv: Connection reset by peer (104)
Total of 1 requests completed
sam@MEX8G7FJG3:~/inbound$ ab -n 4096 -c 4096 http://localhost:9999/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
apr_socket_recv: Connection refused (111)

the port closes and debug messages shows
time="2025-10-30T16:17:29-06:00" level=debug msg="[rootlesskit:parent] port/builtin: accept: accept tcp4 0.0.0.0:9999: accept4: too many open files"
time="2025-10-30T16:17:29-06:00" level=debug msg="[rootlesskit:parent] port/builtin: copyConnToChild: file file+net : fcntl: too many open files"