tasks failing with no logs

[aamster]

Apache Airflow version

2.5.1

What happened

tasks are sporadically failing with no logs

tried following https://cloud.google.com/composer/docs/how-to/using/troubleshooting-dags#task_fails_without_emitting_logs by increasing dagbag-import-timeout, dag-file-processor-timeout

unfortunately still get same problem

I don't see any indication of a problem in any log

What you think should happen instead

No response

How to reproduce

dunno

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Yopu should look at your deploynment (i.e kubernetes) for detailed logs. My bet is (since you are using Mac and Helm). Your docker has too little memory to run Airflow (we explain in https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#running-airflow-in-docker) that the default setting on Mac is to small to run Airflow in docker-compose, I guess it's the same for running K8S.

But that's a wild guess and you should look at the logs of your k8s the same as for any other apps you run there. K8S has many layers and not all logs will be surfaced to the applicaiton layer, when you run any application on K8S, you need to learn and fine the right tools for you to monitor k8s, not only airlfow on top of k8s as the healthiness and resources of K8S platform as a whole might influence the applications that are run. I recommend k9s - and if you want to be a deployment manager and you choose to r run and deploy anything on k8S you should master and learn how to monitor those - there is no escape from that.

[potiuk]

If you find more logs indicating the problems - it would be great if you can post them here for others to learn, having discussions like that is a useful resource for others when they are struggiling with the same issue. Airflow is run by 2500 and they are helping each other for free, the software is free as well, so giving back by posting such information is great as there is a chance someone looking for similar issues will not have to solve it on their own.

[aamster]

[potiuk]

Then please provide more information that will help anyone looking at it to solve the issue.

Again - you completely misjugdge the role of issues here. This is not a helpdesk who you can expect to solve your troubleshooting problems. Airflow is created. by 2500 volunteers, to produce software that you can use for free. If you expect those people to work their nights and weekends to solve your problem with using the software you paid exactly 0 for, then it means that you want to effectively to get those people to work for you for free.

On the other hand if you spend time and provide more information to help those volunteers to help you to solve your problem, then you might become a valuable member of the community.

It's your choice to make.

[aamster]

Great job, thanks for the help!!!! I really appreciate you spending the time to help me with this issue. Thanks so much !! Have a great one!

[potiuk]

I also appreciate your time to give back to the community for the free software you got. This is fantastic help to the community!

[potiuk]

I especially appreciate the meticulous details and the time you spent to try to help the community to diagnose and help others (those who spend time on analysing and helping others).

The level of details you provided is exceptional, comparing to other reports. It's so easy to diagnose and read it. It takes no time at all to be able to help you. I wish everyone else provided so much detaild and have shown that they've "done their part". Just look and compare, this one is really an exceptional one that does not compare to all other reports.