Different retention periods by tenant are not working as expected

[KrishnaJyothika]

What is the bug?

Hi,

Different retention periods by tenant are not working as expected. Despite updating the runtime configuration, the retention policy still defaults to 30 days for all tenants.

overrides:
  tenant:
    compactor_blocks_retention_period: 60d

Runtime Configuration: The retention periods appear updated in the /overrides-exporter/runtime_config?mode=diff API.

Cortex Limits Metric: The cortex_limits_overrides metric does not reflect the updated retention periods under limit_name. (Is this expected? can see for other limits for defined tenants but not on retention period)

Storage Account: Only the last 30 days of blocks are present for the tenants, indicating that the default retention period (30 days) is still being used.

How to reproduce it?

Configure different retention periods for two tenants in the runtime configuration.

Check the /overrides-exporter/runtime_config?mode=diff API to confirm the updates.

Observe the cortex_limits_overrides metric to see if the retention periods are reflected.

Check the storage account to see which blocks are present for each tenant.

What did you think would happen?

Followed according to below document
https://grafana.com/docs/mimir/latest/configure/configure-metrics-storage-retention/

What was your environment?

Infrastructure - Kubernetes
Deployment - helm

Any additional context to share?

Mimir version - 2.14.2

[KrishnaJyothika]

Further debugging on this, seems like there is no issue from runtime configuration. Can see retention period is properly getting picked up for different tenants but the data blocks are getting deleted from the backend storage.

Compactor logs
ts=2024-12-11T10:37:56.080414097Z caller=blocks_cleaner.go:629 level=info component=cleaner run_id=123456 task=clean_up_users user=tenant msg="marked blocks for deletion" num_blocks=0 retention=1440h0m0s

Requesting help to fix this issue

[mcg-aleksi-walden]

Can we get some update on this one? No matter what combination I try with limits and overrides on the compactor_blocks_retention_period when using the mimir-distributed helm chart, it always uses the value configured in the limits.

mimir:
  structuredConfig:
    limits:
      compactor_blocks_retention_period: 1d

runtimeConfig:
  overrides:
    tenant-a:
      compactor_blocks_retention_period: 2d
    tenant-b:
      compactor_blocks_retention_period: 1d

This configuration still sets the retention for tenant-a to 1 days

[deniszh]

As far as I understand, global limit is max, and tenants limits can be lower but not higher.

[mcg-aleksi-walden]

That would be a bit unfortunate in a scenario where you have multiple tenants and need to increase retention in only one. That would mean that for every new tenant and existing one you'd have to declare a new tenant block into the overrides blocks to decrease the retention to "normal" level.

Also i don't spot any mention of only being able to have lower value in the overrides compared to limits here. https://grafana.com/docs/mimir/latest/configure/about-runtime-configuration/#runtime-configuration-of-per-tenant-limits. Quite the contrary. The examples show higher values vs limits