Different retention periods by tenant are not working as expected #10527
Replies: 3 comments 1 reply
-
|
Further debugging on this, seems like there is no issue from runtime configuration. Can see retention period is properly getting picked up for different tenants but the data blocks are getting deleted from the backend storage. Compactor logs Requesting help to fix this issue |
Beta Was this translation helpful? Give feedback.
-
|
Can we get some update on this one? No matter what combination I try with limits and overrides on the This configuration still sets the retention for tenant-a to 1 days |
Beta Was this translation helpful? Give feedback.
-
|
As far as I understand, global limit is max, and tenants limits can be lower but not higher. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
What is the bug?
Hi,
Different retention periods by tenant are not working as expected. Despite updating the runtime configuration, the retention policy still defaults to 30 days for all tenants.
Runtime Configuration: The retention periods appear updated in the /overrides-exporter/runtime_config?mode=diff API.
Cortex Limits Metric: The cortex_limits_overrides metric does not reflect the updated retention periods under limit_name. (Is this expected? can see for other limits for defined tenants but not on retention period)
Storage Account: Only the last 30 days of blocks are present for the tenants, indicating that the default retention period (30 days) is still being used.
How to reproduce it?
Configure different retention periods for two tenants in the runtime configuration.
Check the /overrides-exporter/runtime_config?mode=diff API to confirm the updates.
Observe the cortex_limits_overrides metric to see if the retention periods are reflected.
Check the storage account to see which blocks are present for each tenant.
What did you think would happen?
Followed according to below document
https://grafana.com/docs/mimir/latest/configure/configure-metrics-storage-retention/
What was your environment?
Infrastructure - Kubernetes
Deployment - helm
Any additional context to share?
Mimir version - 2.14.2
Beta Was this translation helpful? Give feedback.
All reactions