Enable different log level for airflow classes than the one defined globally

[llamageddon83]

Apache Airflow version

2.5.3

What happened

We are running into an issue where metrics in Grafana show executor running tasks do not get cleared and over time accumulate. It results in Airflow progressively slowing down where we need to restart scheduler every 2-3 weeks and things go back to normal. However, before I open that bug, I wanted to make sure that it is not something related to a DAG implementation. And I would like to get one specific debug log out of kuberenetes_exector.py that logs the list of tasks running:

           self.log.debug("self.running: %s", self.running)

That code is located here. Since I use LocalExecutor locally, I tried outputting debug logs from another class as well. (Please take a look at the logging configuration snippet below).

Long story short, I am unable to configure debug logging only for these classes. The global log level setting of INFO is being applied. Unless the global setting is changed to DEBUG, debug logs are not outputted. And setting global level to DEBUG is bit tricky because that ends up generating too many logs in production.

The log configuration is:

from copy import deepcopy

# import the default logging configuration
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG

LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)

LOGGING_CONFIG['loggers']['airflow_executors_kubernetes_executor'] = {
    "class": "airflow.executors.kubernetes_executor",
    "handlers": ["console"],
    "level": 'DEBUG',
    "propagate": True,
}

LOGGING_CONFIG['loggers']['airflow_jobs_scheduler_job'] = {
    "class": "airflow.jobs.scheduler_job",
    "handlers": ["console"],
    "level": 'DEBUG',
    "propagate": True,
}

I have validated that the log configuration is loaded correctly by looking at the output of cli command airflow config

[logging]
...
logging_config_class = config.log_config.LOGGING_CONFIG
...

I can also see the following line in the logs:

Successfully imported user-defined logging config from config.log_config.LOGGING_CONFIG

What you think should happen instead

The DEBUG logs from the specified class in the logging configuration should be outputted rather than the global setting being applied.

How to reproduce

Configure custom logging. This setup may vary for everyone. I set it up in a dockerfile like this:

FROM apache/airflow:slim-2.5.3-python3.9

COPY requirements.txt ./

RUN pip install -r requirements.txt --no-cache-dir --use-deprecated=legacy-resolver

USER root

RUN sudo apt-get update && apt-get install sudo && apt-get install net-tools && apt-get -y install smbclient


RUN chmod -R 777 /opt/airflow/decrypt && chown -R airflow /opt/airflow/

USER airflow

ENV PYTHONPATH=/opt/airflow/dags

#The following is required for custom loggers.
RUN mkdir $PYTHONPATH/config
COPY log_config.py $PYTHONPATH/config/
RUN touch $PYTHONPATH/config/__init__.py
ENV AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=config.log_config.LOGGING_CONFIG

Operating System

MacOS Ventura 13.1

Versions of Apache Airflow Providers

The requirements.txt file used in the above dockerfile

This file contains the dependencies required by DAGs and is used to build the Airflow docker image.

Alphabetical

aiofiles==23.1.0
aiohttp==3.8.4
airflow-dbt>=0.4.0
airflow-exporter==1.5.3
anytree==2.8.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http>=2.0.3
apache-airflow-providers-microsoft-mssql==2.1.3
apache-airflow-providers-snowflake>=4.0.4
apache-airflow-providers-hashicorp==3.3.0
apache-airflow-providers-cncf-kubernetes==5.2.2
apache-airflow>=2.2.3
asgiref==3.5.0
Authlib==0.15.5
dbt-snowflake==1.2.0
flatdict==4.0.1
hvac==0.11.2
jsonschema>=4.17.3
pandas==1.3.5
psycopg2-binary==2.9.3
pyOpenSSL==23.1.1
pysftp==0.2.9
pysmbclient==0.1.5
python-gnupg==0.5.0
PyYAML~=5.4.1
requests~=2.26.0
smbprotocol==1.9.0
snowflake-connector-python== 3.0.2
snowflake-sqlalchemy==1.4.7
statsd==3.3.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

I am using Airflow provided docker-compose file. It can be found here

The Airflow version I am using is 2.5.3.

Anything else

Every time.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This is very, very, very, very niche case.

I suggest in this case you simply patch the class in question and change log level to info in your image that should be a simple SED command to replace specific log.debug into log.info

If you want to discuss it further - feel free, but complexity to configure something like you want in a configurable way, comparing to you patching single class is incomparable.

[llamageddon83]

Hmm makes sense. I guess I can also copy kuberenetes_executor.py file, update log levels and pass it in as a custom executor to Airflow. I will give both of these a shot. Thank you 🙏