'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker

[muzehyun]

Running airflow 3.0.6 with docker.
It works okay without docker.

Config: airflow-config.txt
docker-compose.yaml.txt

dag

import datetime

from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator


def foo(**kwargs):
    return "Whatever you return gets printed in the logs"


with DAG(
    dag_id="python_operator_dag",
    start_date=datetime.datetime(2025, 1, 1),
    schedule=None,
):
    PythonOperator(task_id="foo", python_callable=foo)

Errors from worker log

 -------------- celery@1df315592f1d v5.5.3 (immunity)
--- ***** -----
-- ******* ---- Linux-6.15.11-orbstack-00539-g9885ebd8e3f4-aarch64-with-glibc2.31 2025-09-08 16:55:52
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         airflow.providers.celery.executors.celery_executor:0xffff826b8100
- ** ---------- .> transport:   redis://redis:6379/1
- ** ---------- .> results:     postgresql://airflow:**@postgres:5432/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> default          exchange=default(direct) key=default


[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . execute_workload

2025-09-08 16:55:53.936632 [info     ] Connected to redis://redis:6379/1 [celery.worker.consumer.connection]
2025-09-08 16:55:53.937856 [info     ] mingle: searching for neighbors [celery.worker.consumer.mingle]
2025-09-08 16:55:54.946640 [info     ] mingle: all alone              [celery.worker.consumer.mingle]
2025-09-08 16:55:59.967371 [info     ] celery@1df315592f1d ready.     [celery.apps.worker]
2025-09-08 16:56:04.760294 [info     ] Task execute_workload[6138f6e9-d8f5-4344-a81c-34a97e19c299] received [celery.worker.strategy]
2025-09-08 16:56:04.817455 [info     ] [6138f6e9-d8f5-4344-a81c-34a97e19c299] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('01992a41-924a-70ba-8e75-91d168df4936'), task_id='foo', dag_id='python_operator_dag', run_id='manual__2025-09-08T16:56:04.394367+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=1, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('python_operator_dag.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=python_operator_dag/run_id=manual__2025-09-08T16:56:04.394367+00:00/task_id=foo/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-09-08 16:56:04.823034 [info     ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
2025-09-08 16:56:04.830088 [warning  ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:05.837363 [warning  ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:07.154363 [warning  ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:08.187003 [warning  ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:09.901427 [info     ] Process exited                 [supervisor] exit_code=-9 pid=33 signal_sent=SIGKILL
2025-09-08 16:56:09.918525 [error    ] Task execute_workload[6138f6e9-d8f5-4344-a81c-34a97e19c299] raised unexpected: ConnectError('[Errno 111] Connection refused') [celery.app.trace]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:101 in                      │
│ map_httpcore_exceptions                                                                          │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:250 in handle_request       │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py:256 in handle_request  │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py:236 in handle_request  │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py:101 in handle_request       │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py:78 in handle_request        │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py:124 in _connect             │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpcore/_backends/sync.py:207 in connect_tcp            │
│                                                                                                  │
│ /usr/local/lib/python3.10/contextlib.py:153 in __exit__                                          │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py:14 in map_exceptions             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/lib/python3.10/site-packages/celery/app/trace.py:453 in trace_task                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/celery/app/trace.py:736 in __protected_call__            │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/airflow/providers/celery/executors/celery_executor_utils │
│ .py:171 in execute_workload                                                                      │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py:1829 in         │
│ supervise                                                                                        │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py:933 in start    │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py:944 in          │
│ _on_child_started                                                                                │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/api/client.py:152 in start                   │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:1218 in patch                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:338 in wrapped_f                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:477 in __call__                     │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:378 in iter                         │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:420 in exc_check                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:187 in reraise                      │
│                                                                                                  │
│ /usr/local/lib/python3.10/concurrent/futures/_base.py:451 in result                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/concurrent/futures/_base.py:403 in __get_result                        │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:480 in __call__                     │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/api/client.py:735 in request                 │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:825 in request                          │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:914 in send                             │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:942 in _send_handling_auth              │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:979 in _send_handling_redirects         │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:1014 in _send_single_request            │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:249 in handle_request       │
│                                                                                                  │
│ /usr/local/lib/python3.10/contextlib.py:153 in __exit__                                          │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:118 in                      │
│ map_httpcore_exceptions                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ConnectError: [Errno 111] Connection refused

[potiuk]

You seem to mix webserver (airflow 2) and apiserver (airflow 3). your config and docker has webserver config, not api -server config - suggest you generate fresh config from airflow 3 and only apply changes you want to apply rather than copying config from airflow 2 - also you can look at the quick start docker-compose of ours (you will find it in a doc) for an inspiration for the api_server change in Airflow 3 and how to adapt your compose file.

Generally speaking you have something wrongly configured so that the worker cannot connect to the api-server (port or hostname)

[jibrilhp]

Hi, I see you're having connection issues. I'm using a Kubernetes environment and solved my problem by changing the AIRFLOW__CORE__EXECUTION_API_SERVER_URL. I'm not sure if it applies to your exact case, but it might be worth checking if your worker can properly connect to the API server.

[Tzinch21]

Faced with the same error.
Check that your environment variable refers to your api_server in docker
For example:
AIRFLOW__CORE__EXECUTION_API_SERVER_URL=http://api-server:8080/execution/