S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3

[chyzouakia]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.29.0

Apache Airflow version

2.10.5

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployment through the Helm chart https://airflow-helm.github.io/charts version 8.9.0.
Airflow image : slim-2.10.5-python3.10
My deployment creates a Service Account to allow to connect to our AWS S3 to push the logs there.

Configuration for logging :
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://{{ bucket }}/airflow/logs"
AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "s3_remote_logging"

What happened

It seems the close() method in S3TaskHandler is not called. I don't have any log stored in S3.
I am able to read a file from S3 :

I am able to push a log from the worker manually by running this code :

from airflow.providers.amazon.aws.hooks.s3 import S3Hook
s3_hook=S3Hook(aws_conn_id="s3_remote_logging")
import os
import pathlib
log = pathlib.Path([LOCAL_PATH]).read_text()
s3_hook.load_string(log, key=[S3_KEY], replace=True, encrypt=False)

What you think should happen instead

When closing the task, The Task Handler should have pushed the logs to my S3 bucket.

How to reproduce

• Run the Airflow image slim-2.10.5-python3.10 with the amazon provider package
• Configure Airflow to push the logs to S3 (the connection should be created) :
  • AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://{{ bucket }}/airflow/logs"
  • AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
  • AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "s3_remote_logging"
• Run a task and see if the code pushes the logs to S3.

Anything else

This problem is occurring on all my tasks now. Please let me know if I misconfigured something. Thanks !

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

My guess is something wrong with the chart, maybe the variables are not passed as you think they are - but you should check there - this chart is not managed by us. This works if you configure it locally in general (a lot of people would complain if not) - so something must be problematic in the configuration of the chart, or you have some typo. Generally speaking you should find information in the local logs (writtent in the place where the chart configures them to be written) and you should look there for clues.

[potiuk]

Converting to a discussion.

[chyzouakia]

I tried in local with Docker and was not able to push the logs to S3 either 🙂 The variables are OK since I am able to read a log file I pushed manually in the S3 bucket (Screenshot shared above).
What local logs are you referring to ? I couldn't find any logs, I tried all the exception messages in the logs and couldn't find one

[potiuk]

was not able to push the logs to S3 either 🙂

I think it would be good to look why -> if there is a reason you could not push it manually, it should be also the same reason logs are not sent by Airflow. You need to see what the problem is.

What local logs are you referring to ? I couldn't find any logs, I tried all the exception messages in the logs and couldn't find one

Any logs produced in the logs folder of Airflow - see "logging" docs in Airflow documentation.

[chyzouakia]

I was able to push the logs from the worker and from local manually. Everything is explained in the description above, I could run the code that pushes the logs manually. I was not able to push the logs to S3 using the S3TaskHandler (with s3 logging configured). I tried checking the logs and couldn't find any error.
It feels like you didn't check the what happened section above and just dismissed the issue 🙂
All my tasks when trying to read in S3 the logs return No logs found on s3 :

I don't have any logs pushed into S3 automatically which lead me to believe there is an issue with the close() method in S3TaskHandler because :

• When I connect to any worker in my cluster, I am able to push logs to the S3 manually using the service account and the S3 remote logging connection in Airflow (The code is in the what happened section)
• When I push the logs manually to S3 the task is able to read the logs in S3 :