Replies: 6 comments 15 replies
-
|
It will be rather hard to diagnose it without more information and logs. I think in order to be able to diagnose it we need more information - first of all more details on your deployment. Which database version you use ? Do you have locking behaviour disabled ? (It would seem so) There should be no case that two schedulers should send tasks for teh same task/dag run if locking is in place. At any case - in order to be able to investigate it we will need detailed and full logs of both schedulers from around the time it happens, and idealy also logs from the database server from around the same time - they shoudl show locks being acquired and released. Converted to discussion until more information provided. |
Beta Was this translation helpful? Give feedback.
-
|
We experienced a similar symptom (task log 1, task log 2)with "one" scheduler when attempting to run airflow scheduler with a fixed number of loops (e.g. start it up with |
Beta Was this translation helpful? Give feedback.
-
|
I experienced too. My airflow is
I experienced 6 times in 3 days. 😢 Logs airflow logs (try 1) airflow log (try 2 at same time) scheduler log from around the time it happens for task |
Beta Was this translation helpful? Give feedback.
-
|
I am experiencing same issue. Airflow scheduler is retrying task when first one still running. My airflow setup is as below Version: 2.2.4 DAG - a simple DAG which prints few lines, sleeps, print few lines, sleeps... Logs: Executor DAG Try - 1: DAG Try - 2: |
Beta Was this translation helpful? Give feedback.
-
|
We encountered this issue today on Airflow 2.4.3 running on MWAA. Here's how things went down for us: 8:13 UTC: attempt 1 of the task begins We are going to mitigate by tuning our However these solutions only make it less likely that this issue will occur; ideally I would really like a solution that guarantees that two attempts won't run concurrently, as our tasks are not designed to handle concurrency gracefully, and chaotic things can happen if this is allowed. |
Beta Was this translation helpful? Give feedback.
-
|
We have been hitting the same issue recently Our airflow setup is as below
After some investigation, I found that the issue happens under a quite rare condition: a new deployment takes place while the current scheduler is executing
With this situation, when the new deployment takes place, a new scheduler is up (its job_id is now 3, for example) and execute Related source code:
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Apache Airflow version
2.2.3
What happened
As far as I can tell, this is the series of events:
Scheduler logs
Messages, truncated/formatted for reading (from flower)
Worker c8
Worker hl
Then, there are two task instance logs:
What you expected to happen
No response
How to reproduce
I can't repro this on demand, but it does happen about once a day specifically for this hourly task.
Operating System
Ubuntu 19.10
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
CeleryKubernetesExecutor
Celery Queue: RabbitMQ
2 schedulers
worker_prefetch_multiplier is not set (defaults to 1)
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions