Issue with connection to pgbouncer

[thegunner157]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

v: 2.6.0

At times, our Airflow instance loses connection to the metadata database during task execution. This makes task execution pod going into Error state.

What you think should happen instead

There should be 3 steps:

• recheck state from metadata DB and clear pod
• remove the pod in error state

How to reproduce

Spin up task that takes a few seconds and kill connection to the metadata DB during execution.

Operating System

Debian GNU/Linux

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.0.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==6.1.0
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-docker==3.6.0
apache-airflow-providers-elasticsearch==4.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-google==10.0.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-microsoft-azure==6.0.0
apache-airflow-providers-microsoft-mssql==3.3.2
apache-airflow-providers-microsoft-psrp==2.2.0
apache-airflow-providers-microsoft-winrm==3.0.0
apache-airflow-providers-mysql==5.0.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-oracle==3.0.0
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==4.2.4
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-snowflake==4.0.5
apache-airflow-providers-sqlite==3.3.2
apache-airflow-providers-ssh==3.6.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployed on AKS

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[hussein-awala]

This makes task execution pod going into Error state.

Is the task state passed to "up_for_retry" or "failed"? Does it get stuck in the some state (ex `queued``)?

When the worker lose the metadata connection (celery or k8s worker), it will not be able to get the task state, push/pull xcom, pull jinja context, update the task state, ... so it's normal that it fails.

I convert it to discussion, and if you really think that there is a bug, you can provide more information about the issue, and we can re-open it if needed.

[thegunner157]

Hi, thank you for your reply.

The task state was passed to "failed" just after those problems with the database.

I wanted to ask if there are (or should be) better end results from Airflow in such cases instead of making the pod lie in an error state

[arjunanan6]

It looks like the configuration way of handling a cleanup would be to use a combination of:

1. https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#delete-worker-pods
2. https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#delete-worker-pods-on-failure

[hussein-awala]

The task state was passed to "failed" just after those problems with the database.

do have a retry > 0 in your task, and despite that, it passes directly to failed?