What is the log groomer sidecar, what does it do, and why does it use so much memory?

[michael-pattern]

Sorry for the basic question, but, I can't seem to find any docs that talk about what the log groomer, neither in the official documentation, nor here on this GitHub repo. Are there any docs for this component currently?

[lsabreu96]

Can't answer on memory consumption, but the sidecar cleans up the *.log files in airflow every now and then

[PauloCarneiro99]

Not sure if this reply 100% of your question but I was taking a look at the helm params reference

Checking this script /clean-logs you can see the cleanup logic

#!/usr/bin/env bash


set -euo pipefail

readonly DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}"
readonly RETENTION="${AIRFLOW__LOG_RETENTION_DAYS:-15}"

trap "exit" INT TERM

readonly EVERY=$((15*60))

echo "Cleaning logs every $EVERY seconds"

while true; do
  echo "Trimming airflow logs to ${RETENTION} days."
  find "${DIRECTORY}"/logs \
    -type d -name 'lost+found' -prune -o \
    -type f -mtime +"${RETENTION}" -name '*.log' -print0 | \
    xargs -0 rm -f || true

  find "${DIRECTORY}"/logs -type d -empty -delete || true

  seconds=$(( $(date -u +%s) % EVERY))
  (( seconds < 1 )) || sleep $((EVERY - seconds - 1))
  sleep 1
done

https://airflow.apache.org/docs/helm-chart/1.8.0/parameters-ref.html

[potiuk]

Likely the "memory" you mention is memory-mapped file memory not the RSS memory, If you do have some evidences this is an actual memory leak open an issue with all details (including type oif memory and the way it grows from your monitoring).