[Bug]: Parallel reapers creating error

[AndreyBaranyuk]

Testcontainers version

v0.38.0

Using the latest Testcontainers version?

Yes

Host OS

MacOS

Host arch

x86

Go version

1.24.4

Docker version

Client:
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:35 2025
 OS/Arch:           darwin/amd64
 Context:           orbstack

Server: Docker Engine - Community
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 11:52:20 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.1.1
  GitCommit:        cb1076646aa3740577fafbf3d914198b7fe8e3f7
 runc:
  Version:          1.3.0
  GitCommit:        4ca628d1d4c974f92d24daccb901aa078aad748e
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    28.2.2
 Context:    orbstack
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0
    Path:     /Users/andrey/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.36.2
    Path:     /Users/andrey/.docker/cli-plugins/docker-compose
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-ai" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-ai: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-debug" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-debug: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-desktop" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-desktop: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-extension" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-extension: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-feedback" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-feedback: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-init" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-init: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-sbom" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-sbom: no such file or directory
WARNING: Plugin "/Users/andrey/.docker/cli-plugins/docker-scout" is not valid: failed to fetch metadata: fork/exec /Users/andrey/.docker/cli-plugins/docker-scout: no such file or directory

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 112
 Server Version: 28.2.2
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: cb1076646aa3740577fafbf3d914198b7fe8e3f7
 runc version: 4ca628d1d4c974f92d24daccb901aa078aad748e
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.10-orbstack-00291-g1b252bd3edea
 Operating System: OrbStack
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.67GiB
 Name: orbstack
 ID: 7327d06a-8a1b-4c62-b058-8a08e13c77d2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
 Default Address Pools:
   Base: 192.168.97.0/24, Size: 24
   Base: 192.168.107.0/24, Size: 24
   Base: 192.168.117.0/24, Size: 24
   Base: 192.168.147.0/24, Size: 24
   Base: 192.168.148.0/24, Size: 24
   Base: 192.168.155.0/24, Size: 24
   Base: 192.168.156.0/24, Size: 24
   Base: 192.168.158.0/24, Size: 24
   Base: 192.168.163.0/24, Size: 24
   Base: 192.168.164.0/24, Size: 24
   Base: 192.168.165.0/24, Size: 24
   Base: 192.168.166.0/24, Size: 24
   Base: 192.168.167.0/24, Size: 24
   Base: 192.168.171.0/24, Size: 24
   Base: 192.168.172.0/24, Size: 24
   Base: 192.168.181.0/24, Size: 24
   Base: 192.168.183.0/24, Size: 24
   Base: 192.168.186.0/24, Size: 24
   Base: 192.168.207.0/24, Size: 24
   Base: 192.168.214.0/24, Size: 24
   Base: 192.168.215.0/24, Size: 24
   Base: 192.168.216.0/24, Size: 24
   Base: 192.168.223.0/24, Size: 24
   Base: 192.168.227.0/24, Size: 24
   Base: 192.168.228.0/24, Size: 24
   Base: 192.168.229.0/24, Size: 24
   Base: 192.168.237.0/24, Size: 24
   Base: 192.168.239.0/24, Size: 24
   Base: 192.168.242.0/24, Size: 24
   Base: 192.168.247.0/24, Size: 24

WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is set

What happened?

Very often, when running pipelines via gitlabci, I see this error:

generic container: create container: reaper: new reaper: run container: container create: Error response from daemon: Conflict. The container name "/reaper_e8facf14dffd094b524bc013704abc9562b31cde768d323e4faf746b6d12268b" is already in use by container "ab31a5113346b84c22c8ffc46a7c424d87289e3f2465407d0dc43620986d707b". You have to remove (or rename) that container to be able to reuse that name.

I use t.Parallel() in my test and as I see, this error is related to parallel containers creating, so that multiple containers are trying to create reaper.

Relevant log output

Additional information

No response

[fedorkanin]

I experience a similar problem in GitLab CI, however I don't use t.Parallel. It seems to appear randomly sometimes, can't provide a repro for now.

Example of the log I'm getting:

Could not create new network: reaper: new reaper: run container: container create: Error response from daemon: Conflict. The container name "/reaper_2aaf442ce96e7a23dd9450aaf80ffeb72658cb77da9cc5814fa60f1f9b057f29" is already in use by container "699b7c9f2b393c6588883dc17783e42b75ed72f849172a31b482f3e454ffe55a". You have to remove (or rename) that container to be able to reuse that name.: failed to create network

[fedorkanin]

We were able to solve the issue with TESTCONTAINERS_RYUK_DISABLED in our CI, because we use DinD and need no container cleanup.

t.Parallel is not a prerequisite for this issue, as tests in different packages always run in parallel. In our case, tests using testcontainers-go in several packages is an essential condition for the issue to occur.

Issue cause

There is an unconfigurable 20s backoff timeout that is used in lookupContainer. When using testcontainers-go in tests of several go packages, lookupContainer should find the healthy reaper if it's already created by another package's tests. If there is no healthy reaper found, a new one is created. If several packages call newReaper, only one of them succeeds (race condition), and the other must retry. On retry, they're gonna call lookupContainer one more time and discover the existing reaper. Relevant issue comment about the retry mechanisms introduced to counter the race condition.

The above is ok until the network doesn't become so slow that the reaper download and start takes more than 20s. Several packages try to download and start it, only one succeeds, and the other ones can't retry because MaxElapsedTime in backoff was reached.

Solution

The relatively easy "fix" from the maintainer's perspective would be to make MaxElapsedTime configurable. It would also be nice to return a more informative error in this case.

From the perspective of the user, it's probably also possible to set TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX and pull ryuk image from their own repo. It's a pity there is no direct WithReaperImage option.