SSL: CERTIFICATE_VERIFY_FAILED in airflow-worker for certificates with internal CA

[frodo2000]

Apache Airflow version

3.0.6

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When API server has setup SSL certificate with local Certificate Authority, airflow-worker is not enable to finish task and service log contains the following error:

Sep 11 13:13:06 dwh-airflow-dev bash[10802]: [2025-09-11 13:13:06 +0000] [10802] [INFO] Handling signal: term Sep 11 13:13:06 dwh-airflow-dev bash[10804]: [2025-09-11 13:13:06 +0000] [10804] [INFO] Worker exiting (pid: 10804) ...skipping... Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ) │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ do = <tenacity.DoAttempt object at 0x7dc38a77b2f0> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = { │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'content': │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ '{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflo… │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'headers': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'cookies': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'follow_redirects': <httpx._client.UseClientDefault object at │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'timeout': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1 │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ } │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ retry_state = <RetryCallState 138278795187296: attempt #5; slept for 6.92; last result: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed (ConnectError [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed: unable to get local issuer certificate (_ssl.c:1000))> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self = <Retrying object at 0x7dc390fe3ad0 (stop=<tenacity.stop.stop_after_attempt │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ object at 0x7dc38a743020>, wait=<retryhttp._wait.wait_context_aware object at │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc38a742ff0>, sleep=<function sleep at 0x7dc38a8e6b60>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ retry=<tenacity.retry.retry_any object at 0x7dc38a742fc0>, before=<function │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ before_nothing at 0x7dc38a8e7d80>, after=<function after_nothing at │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc38a8e6ac0>)> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/airflow/sdk/api/client.py:735 in request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ❱ 735 return super().request(*args, **kwargs) │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ args = ('PATCH', 'task-instances/01993cd3-9124-7f15-9d8a-9f86b86d6c2d/run') │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = { │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'content': │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ '{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflow","'… │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'headers': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'cookies': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'follow_redirects': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'timeout': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1 │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ } │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self = <airflow.sdk.api.client.Client object at 0x7dc38a8517c0> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:825 in request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:914 in send │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:942 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_auth │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:979 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_redirects │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:1014 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_single_request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:249 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ handle_request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /usr/lib/python3.12/contextlib.py:158 in __exit__ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:118 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ map_httpcore_exceptions │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer Sep 12 07:28:54 dwh-airflow-dev bash[27220]: certificate (_ssl.c:1000)

In SDK Client class I found that only server certificate is added to Certifi library:
ctx = ssl.create_default_context(cafile=certifi.where()) if API_SSL_CERT_PATH: ctx.load_verify_locations(API_SSL_CERT_PATH)

Then only self-signed certificates are valid but certificates with local CA (or even intermediate local CA) becomes invalid.
We should consider add additional config parameter like API_SSL_CA_BUNDLE_PATH and add it to certifi context.

Checked workaround:

• adding CA certificated directly to certifi/cacert.pem file - airflow-worker works correctly but each certifi update needs to cacert.pem redefinition
• adding env variables REQUESTS_CA_BUNDLE or SSL_CERT_FILE doesn't work

What you think should happen instead?

No response

How to reproduce

Create SSL certificate with local CA chain

Operating System

Ubuntu

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

Systemd services for:

• postgresql
• redis
• airflow-api
• ariflow-scheduler
• airflow-dag-processor
• airflow-triggerer
• airflow-worker
  with env configuration in /etc/airflow.cfg file
  Celery Executor used

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[emredjan]

Same problem I am having here: #55147

[potiuk]

It seems that what should work for you is to make sure your self-signed certificate is configured as ssl_cert #55147 also - ability to add trust store option has been discussed there, PRs are welcome if that is a better option for you.

Converting to a discussion

[saaifali]

I am facing a similar issue here.
Using Airflow 3.1.1
After Debugging and going through the Airflow code I have noticed the following which I will try my best to explain here :

When creating an HTTP Client in airflow.sdk.api.client.py lines 829 - 832 :

ctx = ssl.create_default_context(cafile=certifi.where())
if API_SSL_CERT_PATH:
    ctx.load_verify_locations(API_SSL_CERT_PATH)
kwargs["verify"] = ctx

Here as we can see, a default Context gets created using certifi.where() which returns : '/etc/ssl/certs/ca-certificates.crt'
It also allows us to load another context into it from the configuration var : API_SSL_CERT_PATH which is the ssl_cert config in [api] in airflow.cfg

The problem is that there are __two completely different SSL configurations in Airflow that serve different purposes, and they're being confused:

1. Server-side SSL Configuration

Location: airflow/cli/commands/api_server_command.py in the _get_ssl_cert_and_key_filepaths() function (lines 147-160)

Purpose: This is for the Airflow API server to serve HTTPS connections (server-side SSL)

Configuration:

• [api] ssl_cert - Server certificate file
• [api] ssl_key - Server private key file

The Restriction: Lines 155-158 enforce that both ssl_cert AND ssl_key must be provided together:
This makes sense because a web server needs both a certificate AND its private key to serve HTTPS.

Now what I want is Client Side verification of the URL which uses the same ssl_cert configuration from above. Which does not really have a ssl_key

The client-side SSL verification code is incorrectly reusing the server-side SSL certificate configuration. The API_SSL_CERT_PATH in client.py is reading from [api] ssl_cert, which is intended for server certificates, not CA certificates for client verification.

hence the ssl_cert has two completely different use cases. And it would be good to move the second use case to use a different config value like ssl_ca_cert or something similar.

Currently all we can do is patch or replace the file returned by certifi.where().

I dont know if my analysis is completely accurate here and I would love to contribute and help fix this up if there is a trust in this solution.
:)