Kafka persistent storage keeps increasing

[adrianmiron]

Hi,

After upgrading to Mimir 3.0 i noticed that the pvc allocated for kafka is slowly filling up with no signs of stopping. The ingesters had a 15G before 3.0 and we never had an alert regarding those volumes. I gave Kafka a 30G volume and it got to almost 20G usage in a few hours.

How is data cleanup up from kafka ? Is there some configuration i missed ?

I did not change the kafka install settings at all, just the auto_create_topic_default_partitions from the 100 in the chart to default(-1)

Thanks

[pracucci]

This looks very similar to the question here. I suggest you to read the linked discussion, but the TL;DR is that you should set a low retention for your Kafka topic.

[adrianmiron]

Yeah, that is the issue right there, the fun part is that that env var is not templated. I will leave my solution here when using ArgoCD in case anyone needs it:

        - group: apps
          kind: StatefulSet
          namespace: <......>
          name: <.....>
          jqPathExpressions:
            - .spec.template.spec.containers[].env[] | select(.name == "KAFKA_LOG_RETENTION_HOURS")