Bug: Ingester use an unusually big amount of memory on restart

[kalEl2001]

What is the bug?

I am currently running Mimir using the mimir-distributed helm chart on a cluster hosted on AWS using spot instances.
After a month, the ingester use a very high amount of memory on restart while replaying WAL after the node hosting the ingester gets preempted.
As of now. I need to scale down the ingesters and delete the PVCs to restore the ingesters.

How to reproduce it?

1. Install Mimir on an EKS cluster using spot instances
2. Wait until the instance get preempted after a month
3. The ingester will use a very big amount of memory until it got killed because of OOM

What did you think would happen?

Ingesters will consume a normal amount of memory and get healthy.

What was your environment?

EKS using spot instances.

Any additional context to share?

No response