How to disable component file logging (scheduler, worker) and log only to stdout/stderr?

[bun-paul-kwon]

Hello Airflow community,

We are currently using AWS MWAA (Celery Executor) and considering migrating to Airflow 3.0.2 on Kubernetes (AWS EKS).

• Image: apache/airflow:3.0.2
• Helm chart: https://airflow.apache.org airflow 1.18.0
• values.yaml

Our task logs are configured to use remote logging to S3, and this is working well. Our configuration(values.yaml) for this is:

executor: "CeleryExecutor"

logging:
  logging_level: "INFO"
  delete_local_logs: "True"
  remote_logging: "True"
  remote_base_log_folder: s3://<MY_AWS_S3_BUCKET>
  remote_log_conn_id: <MY_AIRFLOW_CONNECTION_NAME>

scheduler:
  logGroomerSidecar: false

dagProcessor: 
  logGroomerSidecar: false

workers:
  logGroomerSidecar: false

# ...

Our question is about the logs for Airflow components (e.g., scheduler, dag-processor, worker), not the task logs.

We've observed the following behavior in the Helm chart:

1. The apiserver logs only to stdout/stderr (which is our desired behavior).
2. The scheduler, dag-processor and worker pods log to both stdout/stderr AND local files (in /opt/airflow/logs).
3. Because of this, the logGroomerSidecar is needed to clean up these local files.

Our goal is to disable local file logging for all components (scheduler, worker, dag-processor) and have them log only to stdout/stderr.

We use Datadog autodiscovery, which scrapes these stdout/stderr streams for logging. The local file logs are redundant for us, and we would like to disable the logGroomerSidecar container entirely.

Is there a recommended way to configure this using the Helm chart values.yaml?

Thank you for your guidance!

[bun-paul-kwon]

Hello, @potiuk

Excuse me, I was wondering if you could provide some help.

I'm trying to find out if it's possible to output all logs for scheduler, worker, and dag-processor components directly to stdout/stderr instead of writing to files.

If you have any information or guidance on this, I'd greatly appreciate it! 🙏

Thank you!