2.15.0-rc.0

This release contains 493 PRs from 61 authors, including new contributors Alexander Akhmetov, Daan Schipper, Daniel Kovacs, I. Elisa Pasaoglu, Jay Clifford, Jorge Alberto Díaz Orozco (Akiel), Martin Valiente Ainz, Mia, Michael Tweten, Nikos Angelopoulos, Santi Leira, cui fliter, elsoa-invitech, madhu-reddy-peram. Thank you!

Grafana Mimir version 2.15.0-rc.0 release notes

Grafana Labs is excited to announce version 2.15 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

S2 compression for gRPC is now supported using the following flags:

• -alertmanager.alertmanager-client.grpc-compression=s2
• -ingester.client.grpc-compression=s2
• -querier.frontend-client.grpc-compression=s2
• -querier.scheduler-client.grpc-compression=s2
• -query-frontend.grpc-client-config.grpc-compression=s2
• -query-scheduler.grpc-client-config.grpc-compression=s2
• -ruler.client.grpc-compression=s2
• -ruler.query-frontend.grpc-client-config.grpc-compression=s2

Distributors now support lz4 OTLP compression, and can be deployed in multiple availability zones.

The ruler's <prometheus-http-prefix>/api/v1/rules endpoint now supports exclude_alerts, group_limit, and group_next_token parameters.

mimirtool's analyze ruler/prometheus commands now support bearer tokens.

HTTP client settings can now be tuned for GCS and Azure backends via an http block or corresponding CLI flags.

The compactor now refreshes deletion marks concurrently when updating the bucket index.

The number of Memcached replicas for each type of cache can now be set using configuration settings when using jsonnet.

Important changes

In Grafana Mimir 2.15, the following behavior has changed:

The following alertmanager metrics are not exported for a user when the metric value is zero:

• cortex_alertmanager_alerts_received_total
• cortex_alertmanager_alerts_invalid_total
• cortex_alertmanager_partial_state_merges_total
• cortex_alertmanager_partial_state_merges_failed_total
• cortex_alertmanager_state_replication_total
• cortex_alertmanager_state_replication_failed_total
• cortex_alertmanager_alerts
• cortex_alertmanager_silences

PromQL compatibility has been upgraded from Prometheus 2.0 to 3.0. More details can be found in the Prometheus documentation. The following changes are of note:

• The . pattern in regular expressions in PromQL now matches newline characters.
• Lookback and range selectors are left open and right closed (previously left closed and right closed).
• Native histograms now use exponential interpolation.

Backwards compatibility in dashboards and alerts for thanos_memcached_-prefixed metrics has been removed. These metrics were removed in 2.12 in favor of thanos_cache_-prefixed metrics.

Experimental support for Redis as a cache backend has been removed from jsonnet.

The following deprecated configuration options were removed in this release:

• -distributor.direct-otlp-translation-enabled, which has been enabled by default since 2.13 and is now considered stable.
• -query-scheduler.prioritize-query-components, which is always enabled now
• -ingest-storage.kafka.ongoing-records-per-fetch and -ingest-storage.kafka.startup-records-per-fetch, which have been removed in favour of -ingest-storage.kafka.max-buffered-bytes
• -api.get-request-for-ingester-shutdown-enabled, a deprecated experimental flag which has been marked for removal in 2.15.

Experimental features

Grafana Mimir 2.15 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

Mimir's experimental PromQL engine can now be enabled with -querier.query-engine=mimir. This new engine provides improved performance and reduced querier resource consumption; however, it supports only a subset of all PromQL features. It will fall back to Prometheus' engine for queries containing unsupported features.

The query-frontend can now cache non-transient errors using the experimental flags -query-frontend.cache-errors and -query-frontend.results-cache-ttl-for-errors.

The query-frontend and querier both support an experimental PromQL function, double_exponential_smoothing, which can be enabled by setting -querier.promql-experimental-functions-enabled=true and -query-frontend.promql-experimental-functions-enabled=true.

The ingester can now support out-of-order native histogram ingestion via the flag -ingester.ooo-native-histograms-ingestion-enabled.

The ingester can now build 24h blocks for out-of-order data which is >24h old, using the setting -blocks-storage.tsdb.bigger-out-of-order-blocks-for-old-samples.

The ruler now supports caching the contents of rule groups via the setting -ruler-storage.cache.rule-group-enabled.

The distributor now supports promotion of OTel resource attributes to labels via the setting -distributor.promote-otel-resource-attributes.

Bug fixes

• Alerts: Fix autoscaling metrics joins in MimirAutoscalerNotActive when series churn.
• Alerts: Exclude failed cache "add" operations from alerting since failures are expected in normal operation.
• Alerts: Exclude read-only replicas from IngesterInstanceHasNoTenants alert.
• Alerts: Use resident set memory for the EtcdAllocatingTooMuchMemory alert so that ephemeral file cache memory doesn't cause the alert to misfire.
• Dashboards: Fix autoscaling metrics joins when series churn.
• Distributor: Fix pooling buffer reuse logic when -distributor.max-request-pool-buffer-size is set.
• Ingester: Fix issue where active series requests error when encountering a stale posting.
• Ingester: Fix race condition in per-tenant TSDB creation.
• Ingester: Fix race condition in exemplar adding.
• Ingester: Fix race condition in native histogram appending.
• Ingester: Fix bug in concurrent fetching where a failure to list topics on startup would cause to use an invalid topic ID (0x00000000000000000000000000000000).
• Ingester: Fix data loss bug in the experimental ingest storage when a Kafka Fetch is split into multiple requests and some of them return an error.
• Ingester: Fix bug where chunks could have one unnecessary zero byte at the end.
• OTLP: Support integer exemplar value type.
• OTLP receiver: Preserve colons and combine multiple consecutive underscores into one when generating metric names in suffix adding mode (-distributor.otel-metric-suffixes-enabled).
• Prometheus: Fix issue where negation of native histograms (e.g. -some_native_histogram_series) did nothing.
• Prometheus: Always return unknown hint for first sample in non-gauge native histograms chunk to avoid incorrect counter reset hints when merging chunks from different sources.
• Prometheus: Ensure native histograms counter reset hints are corrected when merging results from different sources.
• PromQL: Fix issue where functions such as rate() over native histograms could return incorrect values if a float stale marker was present in the selected range.
• PromQL: round now removes the metric name again.
• PromQL: Fix issue where metric might not be a counter, name does not end in _total/_sum/_count/_bucket annotation would be emitted even if rate or increase did not have enough samples to compute a result.
• Querier: Fix the behaviour of binary operators between native histograms and floats.
• Querier: Fix stddev+stdvar aggregations to always ignore native histograms, and to treat Infinity consistently.
• Query-frontend: Fix issue where sharded queries could return annotations with incorrect or confusing position information.
• Query-frontend: Fix issue where downstream consumers may not generate correct cache keys for experimental error caching.
• Query-frontend: Support X-Read-Consistency-Offsets on labels queries too.
• Ruler: Fix issue when using the experimental -ruler.max-independent-rule-evaluation-concurrency feature, where the ruler could panic as it updates a running ruleset or shutdowns.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm charts are released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.15.0-rc.0

Grafana Mimir

• [CHANGE] Alertmanager: the following metrics are not exported for a given user when the metric value is zero: #9359
  • cortex_alertmanager_alerts_received_total
  • cortex_alertmanager_alerts_invalid_total
  • cortex_alertmanager_partial_state_merges_total
  • cortex_alertmanager_partial_state_merges_failed_total
  • cortex_alertmanager_state_replication_total
  • cortex_alertmanager_state_replication_failed_total
  • cortex_alertmanager_alerts
  • cortex_alertmanager_silences
• [CHANGE] Distributor: Drop experimental -distributor.direct-otlp-translation-enabled flag, since direct OTLP translation is well tested at this point. #9647
• [CHANGE] Ingester: Change -initial-delay for circuit breakers to begin when the first request is received, rather than at breaker activation. #9842
• [CHANGE] Query-frontend: apply query pruning before query sharding instead of after. #9913
• [CHANGE] Ingester: remove experimental flags -ingest-storage.kafka.ongoing-records-per-fetch and -ingest-storage.kafka.startup-records-per-fetch. They are removed in favour of -ingest-storage.kafka.max-buffered-bytes. #9906
• [CHANGE] Ingester: Replace cortex_discarded_samples_total label from sample-out-of-bounds to sample-timestamp-too-old. #9885
• [CHANGE] Ruler: the /prometheus/config/v1/rules do...

2.13.1

This release contains 9 PRs from 4 authors. Thank you!

Changelog

2.13.1

Grafana Mimir

• [BUGFIX] Upgrade Go to 1.22.9 to address CVE-2024-34156. #10097
• [BUGFIX] Update module google.golang.org/grpc to v1.64.1 to address GHSA-xr7q-jx4m-x55m. #8717
• [BUGFIX] Upgrade github.com/rs/cors to v1.11.0 address GHSA-mh55-gqvf-xfwm. #8611

All changes in this release: mimir-2.13.0...mimir-2.13.1

2.14.2

This release contains 3 PRs from 2 authors. Thank you!

Changelog

2.14.2

Grafana Mimir

• [BUGFIX] Query-frontend: Do not break scheduler connection on malformed queries. #9833

All changes in this release: mimir-2.14.1...mimir-2.14.2

2.14.1

This release contains 2 PRs from 2 authors. Thank you!

Changelog

2.14.1

Grafana Mimir

• [BUGFIX] Update objstore library to resolve issues observed for some S3-compatible object stores, which respond to StatObject with Range incorrectly. #9625

All changes in this release: mimir-2.14.0...mimir-2.14.1

2.14.0

This release contains 599 PRs from 66 authors, including new contributors Adrian Berger, Albert Kerr, Alexander Davis, Alyssa Wada, Aofei Sheng, Bailhache Pierre, Bradley, David Stevens, Davin Kevin, Dennis Haney, Felipe Ferreira, Jeongseup, Nicholas Kress, Paul Farver, Pooya, Rajguru, Sephia Laureencia, Sviat Loginov, Taehyun Kim, Taylor C, Tito Lins, Willem Gillis, William Travis Holton, William Wernert, Yuri Tseretyan. Thank you!

Grafana Mimir version 2.14.0 release notes

Grafana Labs is excited to announce version 2.14 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release. For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

The streaming of chunks from store-gateways to queriers is now enabled by default. This reduces the memory usage in queriers. This was an experimental feature since Mimir 2.10, and is now considered stable.

Compactor adds a new cortex_compactor_disk_out_of_space_errors_total counter metric that tracks how many times a compaction fails due to the compactor being out of disk.

The distributor now replies with the Retry-After header on retriable errors by default. This protects Mimir from clients, including Prometheus, that default to retrying very quickly, making recovering from an outage easier. The feature was originally added as experimental in Mimir 2.11.

Incoming OTLP requests were previously size-limited with the distributor's -distributor.max-recv-msg-size configuration.
The distributor has a new -distributor.max-otlp-request-size configuration for limiting OTLP requests. The default value is 100 MiB.

Ingesters can be marked as read-only as part of their downscaling procedure. The new prepare-instance-ring-downscale endpoint updates the read-only status of an ingester in the ring.

Important changes

In Grafana Mimir 2.14, the following behavior has changed:

When running a remote read request, the querier honors the time range specified in the read hints.

The default inactivity timeout of active series in ingesters, controlled by the -ingester.active-series-metrics-idle-timeout configuration, is increased from 10m to 20m.

The following features of store-gateway are changed: -blocks-storage.bucket-store.max-concurrent-queue-timeout is set to five seconds; -blocks-storage.bucket-store.index-header.lazy-loading-concurrency-queue-timeout is set to five seconds; -blocks-storage.bucket-store.max-concurrent is set to 200;

The experimental support for Redis caching is now deprecated and set to be removed in the next major release. Users are encouraged
to switch to use Memcached.

The following deprecated configuration options were removed in this release:

• The -ingester.return-only-grpc-errors option in the ingester
• The -ingester.client.circuit-breaker.* options in the ingester
• The -ingester.limit-inflight-requests-using-grpc-method-limiter option in the ingester
• The -ingester.client.report-grpc-codes-in-instrumentation-label-enabled option in the distributor and ruler
• The -distributor.limit-inflight-requests-using-grpc-method-limiter option in the distributor
• The -distributor.enable-otlp-metadata-storage option in the distributor
• The -ruler.drain-notification-queue-on-shutdown option in the ruler
• The -querier.max-query-into-future option in the querier
• The -querier.prefer-streaming-chunks-from-store-gateways option in the querier and the store-gateway
• The -query-scheduler.use-multi-algorithm-query-queue option in the querier-scheduler
• The YAML configuration frontend.align_queries_with_step in the query-frontend

Experimental features

Grafana Mimir 2.14 includes some features that are experimental and disabled by default. Use these features with caution and report any issues that you encounter:

The ingester added an experimental -ingester.ignore-ooo-exemplars configuration. When set, out-of-order exemplars are no longer reported to the remote write client.

The querier supports the experimental limitk() and limit_ratio() PromQL functions. This feature is disabled by default, but you can enable it with the -querier.promql-experimental-functions-enabled=true setting in the query-frontend and the querier.

Bug fixes

• Alertmanager: fix configuration validation gap around unreferenced templates.
• Alertmanager: fix goroutine leak when stored configuration fails to apply and there is no existing tenant alertmanager.
• Alertmanager: fix receiver firewall to detect 0.0.0.0 and IPv6 interface-local multicast address as local addresses.
• Alertmanager: fix per-tenant silence limits not reloaded during runtime.
• Alertmanager: fix bugs in silences that could cause an existing silence to expire/be deleted when updating the silence fails. This could happen when the updated silence was invalid or exceeded limits.
• Alertmanager: fix help message for utf-8-strict-mode.
• Compactor: fix a race condition between different compactor replicas that may cause a deleted block to be referenced as non-deleted in the bucket index.
• Configuration: multi-line environment variables are flattened during injection to be compatible with YAML syntax.
• HA Tracker: store correct timestamp for the last-received request from the elected replica.
• Ingester: fix the sporadic not found error causing an internal server error if label names are queried with matchers during head compaction.
• Ingester, store-gateway: fix case insensitive regular expressions not correctly matching some Unicode characters.
• Ingester: fixed timestamp reported in the "the sample has been rejected because its timestamp is too old" error when the write request contains only histograms.
• Query-frontend: fix -querier.max-query-lookback and -compactor.blocks-retention-period enforcement in query-frontend when one of the two is not set.
• Query-frontend: "query stats" log includes the actual status_code when the request fails due to an error occurring in the query-frontend itself.
• Query-frontend: ensure that internal errors result in an HTTP 500 response code instead of a 422 response code.
• Query-frontend: return annotations generated during evaluation of sharded queries.
• Query-scheduler: fix a panic in request queueing.
• Querier: fix the issue where "context canceled" is logged for trace spans for requests to store-gateways that return no series when chunks streaming is enabled.
• Querier: fix issue where queries can return incorrect results if a single store-gateway returns overlapping chunks for a series.
• Querier: do not return grpc: the client connection is closing errors as HTTP 499.
• Querier: fix issue where some native histogram-related warnings were not emitted when rate() was used over native histograms.
• Querier: fix invalid query results when multiple chunks are merged.
• Querier: support optional start and end times on /prometheus/api/v1/labels, /prometheus/api/v1/label/<label>/values, and /prometheus/api/v1/series when max_query_into_future: 0.
• Querier: fix issue where both recently compacted blocks and their source blocks can be skipped during querying if store-gateways are restarting.
• Ruler: add support for draining any outstanding alert notifications before shutting down. Enable this setting with the -ruler.drain-notification-queue-on-shutdown=true CLI flag.
• Store-gateway: fixed a case where, on a quick subsequent restart, the previous lazy-loaded index header snapshot was overwritten by a partially loaded one.
• Store-gateway: store sparse index headers atomically to disk.
• Ruler: map invalid org-id errors to the 400 status code.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm charts are released independently. Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.14.0

Grafana Mimir

• [CHANGE] Update minimal supported version of Go to 1.22. #9134
• [CHANGE] Store-gateway / querier: enable streaming chunks from store-gateways to queriers by default. #6646
• [CHANGE] Querier: honor the start/end time range specified in the read hints when executing a remote read request. #8431
• [CHANGE] Querier: return only samples within the queried start/end time range when executing a remote read request using "SAMPLES" mode. Previously, samples outside of the range could have been returned. Samples outside of the queried time range may still be returned when executing a remote read request using "STREAMED_XOR_CHUNKS" mode. #8463
• [CHANGE] Querier: Set minimum for -querier.max-concurrent to four to prevent queue starvation with querier-worker queue prioritization algorithm; values below the minimum four are ignored and set to the minimum. #9054
• [CHANGE] Store-gateway: enabled -blocks-storage.bucket-store.max-concurrent-queue-timeout by default with a timeout of 5 seconds. #8496
• [CHANGE] Store-gateway: enabled -blocks-storage.bucket-store.index-header.lazy-loading-concurrency-queue-timeout by default with a timeout of 5 seconds . #8667
• [CHANGE] Distributor: Incoming OTLP requests were previously size-limited by using limit from -distributor.max-recv-msg-size option. We have added option -distributor.max-otlp-request-size for limiting OTLP requests, with default value of 100 MiB. #8574
• [CHANGE] Distributor: remove metric cortex_distributor_sample_delay_seconds. #8698
• [CHANGE] Query-frontend: Remove deprecated frontend.align_queries_with_step YAML configuration. The configuration option has been moved to per-tenant and default limits since Mimir 2.12. #8733 #8735
• [CHANGE] Store-gateway: Change default of -blocks-storage.bucket-store.max-concurrent to 200. #8768
• [CHANGE] Added new metric `cortex_compactor_disk_ou...

2.14.0-rc.0

This release contains 593 PRs from 66 authors, including new contributors Adrian Berger, Albert Kerr, Alexander Davis, Alyssa Wada, Aofei Sheng, Bailhache Pierre, Bradley, David Stevens, Davin Kevin, Dennis Haney, Felipe Ferreira, Jeongseup, Nicholas Kress, Paul Farver, Pooya, Rajguru, Sephia Laureencia, Sviat Loginov, Taehyun Kim, Taylor C, Tito Lins, Willem Gillis, William Travis Holton, William Wernert, Yuri Tseretyan. Thank you!

Grafana Mimir version 2.14.0-rc.0 release notes

Grafana Labs is excited to announce version 2.14 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

The streaming of chunks from store-gateways to queriers is now enabled by default. This reduces the memory usage in queriers. This was an experimental feature since Mimir 2.10, and is now considered stable.

Compactor adds a new cortex_compactor_disk_out_of_space_errors_total counter metric that tracks how many times a compaction fails due to the compactor being out of disk.

The distributor now replies with the Retry-After header on retryable errors by default. This protects Mimir from clients, including Prometheus, that default to retrying very quickly, making recovering from an outage easier. The feature was originally added as experimental in Mimir 2.11.

Incoming OTLP requests were previously size-limited with the distributor's -distributor.max-recv-msg-size configuration. The distributor has a new -distributor.max-otlp-request-size configuration for limiting OTLP requests. The default value is 100 MiB.

Ingesters can be marked as read-only as part of their downscaling procedure. The new prepare-instance-ring-downscale endpoint updates the read-only status of an ingester in the ring.

Important changes

In Grafana Mimir 2.14, the following behavior has changed:

When running a remote read request, the querier honors the time range specified in the read hints.

The default inactivity timeout of active series in ingesters, controlled by the -ingester.active-series-metrics-idle-timeout configuration, is increased from 10m to 20m.

The following featues of store-gateway are changed: -blocks-storage.bucket-store.max-concurrent-queue-timeout is set to five seconds; -blocks-storage.bucket-store.index-header.lazy-loading-concurrency-queue-timeout is set to five seconds; -blocks-storage.bucket-store.max-concurrent is set to 200.

The experimental support for Redis caching is now deprecated and set to be removed in the next major release. Users are encouraged
to switch to use Memcached.

The following deprecated configuration options were removed in this release:

• The -ingester.return-only-grpc-errors option in the ingester
• The -ingester.client.circuit-breaker.* options in the ingester
• The -ingester.limit-inflight-requests-using-grpc-method-limiter option in the ingester
• The -ingester.client.report-grpc-codes-in-instrumentation-label-enabled option in the distributor and ruler
• The -distributor.limit-inflight-requests-using-grpc-method-limiter option in the distributor
• The -distributor.enable-otlp-metadata-storage option in the distributor
• The -ruler.drain-notification-queue-on-shutdown option in the ruler
• The -querier.max-query-into-future option in the querier
• The -querier.prefer-streaming-chunks-from-store-gateways option in the querier and the store-gateway
• The -query-scheduler.use-multi-algorithm-query-queue option in the querier-scheduler
• The YAML configuration frontend.align_queries_with_step in the query-frontend

Experimental features

Grafana Mimir 2.14 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

The ingester added an experimental -ingester.ignore-ooo-exemplars configuration. When set, out-of-order exemplars are no longer reported to the remote write client.

The querier supports the experimental limitk() and limit_ratio() PromQL functions. This feature is disabled by default,
but you can enable it with the -querier.promql-experimental-functions-enabled=true setting in the query-frontend and the querier.

Bug fixes

• Alertmanager: fix configuration validation gap around unreferenced templates.
• Alertmanager: fix goroutine leak when stored configuration fails to apply and there is no existing tenant alertmanager.
• Alertmanager: fix receiver firewall to detect 0.0.0.0 and IPv6 interface-local multicast address as local addresses.
• Alertmanager: fix per-tenant silence limits not reloaded during runtime.
• Alertmanager: fix bugs in silences that could cause an existing silence to expire/be deleted when updating the silence fails. This could happen when the updated silence was invalid or exceeded limits.
• Alertmanager: fix help message for utf-8-strict-mode.
• Compactor: fix a race condition between different compactor replicas that may cause a deleted block to be referenced as non-deleted in the bucket index.
• Configuration: multi-line environment variables are flattened during injection to be compatible with YAML syntax.
• HA Tracker: store correct timestamp for the last-received request from the elected replica.
• Ingester: fix the sporadic not found error causing an internal server error if label names are queried with matchers during head compaction.
• Ingester, store-gateway: fix case insensitive regular expressions not correctly matching some Unicode characters.
• Ingester: fixed timestamp reported in the "the sample has been rejected because its timestamp is too old" error when the write request contains only histograms.
• Query-frontend: fix -querier.max-query-lookback and -compactor.blocks-retention-period enforcement in query-frontend when one of the two is not set.
• Query-frontend: "query stats" log includes the actual status_code when the request fails due to an error occurring in the query-frontend itself.
• Query-frontend: ensure that internal errors result in an HTTP 500 response code instead of a 422 response code.
• Query-frontend: return annotations generated during evaluation of sharded queries.
• Query-scheduler: fix a panic in request queueing.
• Querier: fix the issue where "context canceled" is logged for trace spans for requests to store-gateways that return no series when chunks streaming is enabled.
• Querier: fix issue where queries can return incorrect results if a single store-gateway returns overlapping chunks for a series.
• Querier: do not return grpc: the client connection is closing errors as HTTP 499.
• Querier: fix issue where some native histogram-related warnings were not emitted when rate() was used over native histograms.
• Querier: fix invalid query results when multiple chunks are merged.
• Querier: support optional start and end times on /prometheus/api/v1/labels, /prometheus/api/v1/label/<label>/values, and /prometheus/api/v1/series when max_query_into_future: 0.
• Querier: fix issue where both recently compacted blocks and their source blocks can be skipped during querying if store-gateways are restarting.
• Ruler: add support for draining any outstanding alert notifications before shutting down. Enable this setting with the -ruler.drain-notification-queue-on-shutdown=true CLI flag.
• Store-gateway: fixed a case where, on a quick subsequent restart, the previous lazy-loaded index header snapshot was overwritten by a partially loaded one.
• Store-gateway: store sparse index headers atomically to disk.
• Ruler: map invalid org-id errors to the 400 status code.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm charts are released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.14.0-rc.0

Grafana Mimir

• [CHANGE] Update minimal supported version of Go to 1.22. #9134
• [CHANGE] Store-gateway / querier: enable streaming chunks from store-gateways to queriers by default. #6646
• [CHANGE] Querier: honor the start/end time range specified in the read hints when executing a remote read request. #8431
• [CHANGE] Querier: return only samples within the queried start/end time range when executing a remote read request using "SAMPLES" mode. Previously, samples outside of the range could have been returned. Samples outside of the queried time range may still be returned when executing a remote read request using "STREAMED_XOR_CHUNKS" mode. #8463
• [CHANGE] Querier: Set minimum for -querier.max-concurrent to four to prevent queue starvation with querier-worker queue prioritization algorithm; values below the minimum four are ignored and set to the minimum. #9054
• [CHANGE] Store-gateway: enabled -blocks-storage.bucket-store.max-concurrent-queue-timeout by default with a timeout of 5 seconds. #8496
• [CHANGE] Store-gateway: enabled -blocks-storage.bucket-store.index-header.lazy-loading-concurrency-queue-timeout by default with a timeout of 5 seconds . #8667
• [CHANGE] Distributor: Incoming OTLP requests were previously size-limited by using limit from -distributor.max-recv-msg-size option. We have added option -distributor.max-otlp-request-size for limiting OTLP requests, with default value of 100 MiB. #8574
• [CHANGE] Distributor: remove metric cortex_distributor_sample_delay_seconds. #8698
• [CHANGE] Query-frontend: Remove deprecated frontend.align_queries_with_step YAML configuration. The configuration option has been moved to per-tenant and default limits since Mimir 2.12. #8733 #8735
• [CHANGE] Store-gateway: Change default of -blocks-storage.bucket-store.max-concurrent to 200. #8768
• [CHANGE] Added new metric `cortex_compa...

2.13.0

This release contains 490 PRs from 67 authors, including new contributors Anthony Keydel, Armand Grillet, AvivGuiser, Daniel R. Dagfinrud, David Collier-Brown, David Grant, Dimitris Alo, Enrique Garbi, Erik Sommer, Ian Halliday, InventiveCoder, Kevin Mingtarja, Lasse Hels, Pangidoan Butar, Quentin Bisson, Rens Groothuijsen, René Gärtner, Ross Brunson, Santiago, Seiya, Spyros Panagiotopoulos, Yuri Nikolic, Zied ABID, kahirokunn, lasermoth. Thank you!

Grafana Mimir version 2.13.0 release notes

Grafana Labs is excited to announce version 2.13 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

• Improved CPU performance in processing queries with regular expressions that contain many alternations (for example foo|bar|baz|...).

• Improved OTLP ingestion performance by using native translation, which reduces memory usage in the distributor by up to 30% and CPU usage by up to 8%.

• Configurable S3 bucket lookup type improves interoperability with S3-compatible providers, such as Tencent and Alibaba. Configure lookup type via -<prefix>.s3.bucket-lookup-type or -common.storage.s3-bucket-lookup-type.

• Configuration of TLS for S3 buckets is now possible via -<prefix>.s3.http.* and -common.storage.s3.http.* configuration options.

• Mimirtool can now verify the validity of a Mimir runtime configuration file with the mimirtool runtime-config verify command.

• Active series are now updated along with owned series. This means the number of active series for a tenant is more accurate after scaling out ingesters. As a result, ingesters now more precisely enforce tenants' series limits. This feature is only enabled when -ingester.track-ingester-owned-series or -ingester.use-ingester-owned-series-for-limits are enabled.

• Store-gateway can be configured to explicitly disable or enable tenants. This is useful in cases where compaction slows down for a tenant to temporarily exclude another tenant while compaction catches up without affecting other tenants.

• Remote read (/prometheus/api/v1/read) is becoming a first-class endpoint in Mimir with support in query stats logs and query blocking in the query-frontend. More coming in future releases.

Additionally, the following previously experimental features are now considered stable:

• Rules tenant federation via the source_tenants field in rule groups.
• Enabling recording, and alerting rules evaluation on a per-tenant basis.
• Limiting the number of tenants a federated query can query.

Important changes

In Grafana Mimir 2.13 the following behavior has changed:

• The default Docker image grafana/mimir is now based on the distroless image gcr.io/distroless/static-debian12.
  See Debugging distroless container images for more details on how to
  work with distroless images.

• Error logs in the ingester are now sampled at 10% by default. Sampled log lines contain (sampled 1/10). The cortex_discarded_samples_total metric still tracks all discarded samples.

• Anonymous usage statistics now include actual CPU usage instead of available CPU cores.

• Continuous-test is no longer a standalone binary and is now part of Mimir as its own target. The published Docker image has been updated to use the new packaging.

The following deprecated configuration options are removed in Grafana Mimir 2.13:

• The configuration option -log.buffered, which was deprecated in 2.11 and is now enabled by default.

Experimental features

Grafana Mimir 2.13 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

• Experimental support for server-side circuit breakers in ingesters on read and write requests. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options and further configured via the -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options.

The following configuration options are deprecated and will be removed in a future Grafana Mimir release:

• evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead.

Bug fixes

• OTLP ingestion: translate all HTTP 5xx errors into one of 502, 503, or 504, so that the otel-collector retries the failed requests.
• OTLP ingestion: return properly formatted protobuf error messages when ingestion fails.
• OTLP ingestion: generate target_info metric only when there are metrics in the request and there is at least one configured identifying label.
• OTLP ingestion: don't discard timeseries paired with invalid exemplars. Instead, try to ingest the timeseries and discard only the invalid exemplars.
• Subqueries: @ end() and @ start() now work correctly with queries split by time.
• Native histograms: order exemplars before ingestion to improve success rate when ingesting multiple exemplars.
• Native histograms: return HTTP 400 on invalid native histogram samples instead of HTTP 500. The metric cortex_discarded_samples_total{reason="invalid-native-histogram"} is now incremented on invalid histogram samples.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm charts are released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.13.0

Grafana Mimir

• [CHANGE] Build: grafana/mimir docker image is now based on gcr.io/distroless/static-debian12 image. Alpine-based docker image is still available as grafana/mimir-alpine, until Mimir 2.15. #8204 #8235
• [CHANGE] Ingester: /ingester/flush endpoint is now only allowed to execute only while the ingester is in Running state. The 503 status code is returned if the endpoint is called while the ingester is not in Running state. #7486
• [CHANGE] Distributor: Include label name in err-mimir-label-value-too-long error message: #7740
• [CHANGE] Ingester: enabled 1 out 10 errors log sampling by default. All the discarded samples will still be tracked by the cortex_discarded_samples_total metric. The feature can be configured via -ingester.error-sample-rate (0 to log all errors). #7807
• [CHANGE] Query-frontend: Query results caching and experimental query blocking now utilize the PromQL string-formatted query format rather than the unvalidated query as submitted to the frontend. #7742
  • Query results caching should be more stable as all equivalent queries receive the same cache key, but there may be cache churn on first deploy with the updated format
  • Query blocking can no longer be circumvented with an equivalent query in a different format; see Configure queries to block
• [CHANGE] Query-frontend: stop using -validation.create-grace-period to clamp how far into the future a query can span. #8075
• [CHANGE] Clamp GOMAXPROCS to runtime.NumCPU. #8201
• [CHANGE] Anonymous usage statistics tracking: add CPU usage percentage tracking. #8282
• [CHANGE] Added new metric cortex_compactor_disk_out_of_space_errors_total which counts how many times a compaction failed due to the compactor being out of disk. #8237
• [CHANGE] Anonymous usage statistics tracking: report active series in addition to in-memory series. #8279
• [CHANGE] Ruler: evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead (it has the same exact meaning and behaviour). #8295
• [CHANGE] General: remove -log.buffered. The configuration option has been enabled by default and deprecated since Mimir 2.11. #8395
• [CHANGE] Ruler: promote tenant federation from experimental to stable. #8400
• [CHANGE] Ruler: promote -ruler.recording-rules-evaluation-enabled and -ruler.alerting-rules-evaluation-enabled from experimental to stable. #8400
• [CHANGE] General: promote -tenant-federation.max-tenants from experimental to stable. #8400
• [FEATURE] Continuous-test: now runable as a module with mimir -target=continuous-test. #7747
• [FEATURE] Store-gateway: Allow specific tenants to be enabled or disabled via -store-gateway.enabled-tenants or -store-gateway.disabled-tenants CLI flags or their corresponding YAML settings. #7653
• [FEATURE] New -<prefix>.s3.bucket-lookup-type flag configures lookup style type, used to access bucket in s3 compatible providers. #7684
• [FEATURE] Querier: add experimental streaming PromQL engine, enabled with -querier.promql-engine=mimir. #7693 #7898 #7899 #8023 #8058 #8096 #8121 #8197 #8230 #8247 #8270 #8276 #8277 #8291 #8303 #8340 #8256 #8348
• [FEATURE] New /ingester/unregister-on-shutdown HTTP endpoint allows dynamic access to ingesters' -ingester.ring.unregister-on-shutdown configuration. #7739
• [FEATURE] Server: added experimental PROXY protocol support. The PROXY protocol support can be enabled via -server.proxy-protocol-enabled=true. When enabled, the support is added both to HTTP and gRPC listening ports. #7698
• [FEATURE] Query-frontend, querier: new experimental /cardinality/active_native_histogram_metrics API to get active native histogram metric names with statistics about active native histogram buckets. #7982 #7986 #...

2.13.0-rc.2

This release contains 476 PRs from 67 authors, including new contributors Anthony Keydel, Armand Grillet, AvivGuiser, Daniel R. Dagfinrud, David Collier-Brown, David Grant, Dimitris Alo, Enrique Garbi, Erik Sommer, Ian Halliday, InventiveCoder, Kevin Mingtarja, Lasse Hels, Pangidoan Butar, Quentin Bisson, Rens Groothuijsen, René Gärtner, Ross Brunson, Santiago, Seiya, Spyros Panagiotopoulos, Yuri Nikolic, Zied ABID, kahirokunn, lasermoth. Thank you!

Grafana Mimir version 2.13.0-rc.2 release notes

Features and enhancements

• Improved CPU performance in processing queries with regular expressions that contain many alternations (for example foo|bar|baz|...).

• Improved OTLP ingestion performance by using native translation, which reduces memory usage in the distributor by up to 30% and CPU usage by up to 8%.

• Configurable S3 bucket lookup type improves interoperability with S3-compatible providers, such as Tencent and Alibaba. Configure lookup type via -<prefix>.s3.bucket-lookup-type or -common.storage.s3-bucket-lookup-type.

• Configuration of TLS for S3 buckets is now possible via -<prefix>.s3.http.* and -common.storage.s3.http.* configuration options.

• Mimirtool can now verify the validity of a Mimir runtime configuration file with the mimirtool runtime-config verify command.

• Active series are now updated along with owned series. This means the number of active series for a tenant is more accurate after scaling out ingesters. As a result, ingesters now more precisely enforce tenants' series limits. This feature is only enabled when -ingester.track-ingester-owned-series or -ingester.use-ingester-owned-series-for-limits are enabled.

• Store-gateway can be configured to explicitly disable or enable tenants. This is useful in cases where compaction slows down for a tenant to temporarily exclude another tenant while compaction catches up without affecting other tenants.

• Remote read (/prometheus/api/v1/read) is becoming a first-class endpoint in Mimir with support in query stats logs and query blocking in the query-frontend. More coming in future releases.

Additionally, the following previously experimental features are now considered stable:

• Rules tenant federation via the source_tenants field in rule groups.
• Enabling recording, and alerting rules evaluation on a per-tenant basis.
• Limiting the number of tenants a federated query can query.

Important changes

In Grafana Mimir 2.13 the following behavior has changed:

• The default Docker image grafana/mimir is now based on the distroless image gcr.io/distroless/static-debian12.
  See Debugging distroless container images for more details on how to
  work with distroless images.

• Error logs in the ingester are now sampled at 10% by default. Sampled log lines contain (sampled 1/10). The cortex_discarded_samples_total metric still tracks all discarded samples.

• Anonymous usage statistics now include actual CPU usage instead of available CPU cores.

• Continuous-test is no longer a standalone binary and is now part of Mimir as its own target. The published Docker image has been updated to use the new packaging.

The following deprecated configuration options are removed in Grafana Mimir 2.13:

• The configuration option -log.buffered, which was deprecated in 2.11 and is now enabled by default.

Experimental features

Grafana Mimir 2.13 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

• Experimental support for server-side circuit breakers in ingesters on read and write requests. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options and further configured via the -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options.

The following configuration options are deprecated and will be removed in a future Grafana Mimir release:

• evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead.

Bug fixes

• OTLP ingestion: translate all HTTP 5xx errors into one of 502, 503, or 504, so that the otel-collector retries the failed requests.
• OTLP ingestion: return properly formatted protobuf error messages when ingestion fails.
• OTLP ingestion: generate target_info metric only when there are metrics in the request and there is at least one configured identifying label.
• OTLP ingestion: don't discard timeseries paired with invalid exemplars. Instead, try to ingest the timeseries and discard only the invalid exemplars.
• Subqueries: @ end() and @ start() now work correctly with queries split by time.
• Native histograms: order exemplars before ingestion to improve success rate when ingesting multiple exemplars.
• Native histograms: return HTTP 400 on invalid native histogram samples instead of HTTP 500. The metric cortex_discarded_samples_total{reason="invalid-native-histogram"} is now incremented on invalid histogram samples.

Changelog

2.13.0-rc.1

Grafana Mimir

• [CHANGE] Build: grafana/mimir docker image is now based on gcr.io/distroless/static-debian12 image. Alpine-based docker image is still available as grafana/mimir-alpine, until Mimir 2.15. #8204 #8235
• [CHANGE] Ingester: /ingester/flush endpoint is now only allowed to execute only while the ingester is in Running state. The 503 status code is returned if the endpoint is called while the ingester is not in Running state. #7486
• [CHANGE] Distributor: Include label name in err-mimir-label-value-too-long error message: #7740
• [CHANGE] Ingester: enabled 1 out 10 errors log sampling by default. All the discarded samples will still be tracked by the cortex_discarded_samples_total metric. The feature can be configured via -ingester.error-sample-rate (0 to log all errors). #7807
• [CHANGE] Query-frontend: Query results caching and experimental query blocking now utilize the PromQL string-formatted query format rather than the unvalidated query as submitted to the frontend. #7742
  • Query results caching should be more stable as all equivalent queries receive the same cache key, but there may be cache churn on first deploy with the updated format
  • Query blocking can no longer be circumvented with an equivalent query in a different format; see Configure queries to block
• [CHANGE] Query-frontend: stop using -validation.create-grace-period to clamp how far into the future a query can span. #8075
• [CHANGE] Clamp GOMAXPROCS to runtime.NumCPU. #8201
• [CHANGE] Anonymous usage statistics tracking: add CPU usage percentage tracking. #8282
• [CHANGE] Added new metric cortex_compactor_disk_out_of_space_errors_total which counts how many times a compaction failed due to the compactor being out of disk. #8237
• [CHANGE] Anonymous usage statistics tracking: report active series in addition to in-memory series. #8279
• [CHANGE] Ruler: evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead (it has the same exact meaning and behaviour). #8295
• [CHANGE] General: remove -log.buffered. The configuration option has been enabled by default and deprecated since Mimir 2.11. #8395
• [CHANGE] Ruler: promote tenant federation from experimental to stable. #8400
• [CHANGE] Ruler: promote -ruler.recording-rules-evaluation-enabled and -ruler.alerting-rules-evaluation-enabled from experimental to stable. #8400
• [CHANGE] General: promote -tenant-federation.max-tenants from experimental to stable. #8400
• [FEATURE] Continuous-test: now runable as a module with mimir -target=continuous-test. #7747
• [FEATURE] Store-gateway: Allow specific tenants to be enabled or disabled via -store-gateway.enabled-tenants or -store-gateway.disabled-tenants CLI flags or their corresponding YAML settings. #7653
• [FEATURE] New -<prefix>.s3.bucket-lookup-type flag configures lookup style type, used to access bucket in s3 compatible providers. #7684
• [FEATURE] Querier: add experimental streaming PromQL engine, enabled with -querier.promql-engine=mimir. #7693 #7898 #7899 #8023 #8058 #8096 #8121 #8197 #8230 #8247 #8270 #8276 #8277 #8291 #8303 #8340 #8256 #8348
• [FEATURE] New /ingester/unregister-on-shutdown HTTP endpoint allows dynamic access to ingesters' -ingester.ring.unregister-on-shutdown configuration. #7739
• [FEATURE] Server: added experimental PROXY protocol support. The PROXY protocol support can be enabled via -server.proxy-protocol-enabled=true. When enabled, the support is added both to HTTP and gRPC listening ports. #7698
• [FEATURE] Query-frontend, querier: new experimental /cardinality/active_native_histogram_metrics API to get active native histogram metric names with statistics about active native histogram buckets. #7982 #7986 #8008
• [FEATURE] Alertmanager: Added -alertmanager.max-silences-count and -alertmanager.max-silence-size-bytes to set limits on per tenant silences. Disabled by default. #8241 #8249
• [FEATURE] Ingester: add experimental support for the server-side circuit breakers when writing to and reading from ingesters. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options. Further -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options for configuring circuit-breaker are available. Added metrics cortex_ingester_circuit_breaker_results_total, `cortex_i...

2.13.0-rc.1

This release contains 476 PRs from 67 authors, including new contributors Anthony Keydel, Armand Grillet, AvivGuiser, Daniel R. Dagfinrud, David Collier-Brown, David Grant, Dimitris Alo, Enrique Garbi, Erik Sommer, Ian Halliday, InventiveCoder, Kevin Mingtarja, Lasse Hels, Pangidoan Butar, Quentin Bisson, Rens Groothuijsen, René Gärtner, Ross Brunson, Santiago, Seiya, Spyros Panagiotopoulos, Yuri Nikolic, Zied ABID, kahirokunn, lasermoth. Thank you!

Grafana Mimir version 2.13.0-rc.0 release notes

Features and enhancements

• Improved CPU performance in processing queries with regular expressions that contain many alternations (for example foo|bar|baz|...).

• Improved OTLP ingestion performance by using native translation, which reduces memory usage in the distributor by up to 30% and CPU usage by up to 8%.

• Configurable S3 bucket lookup type improves interoperability with S3-compatible providers, such as Tencent and Alibaba. Configure lookup type via -<prefix>.s3.bucket-lookup-type or -common.storage.s3-bucket-lookup-type.

• Configuration of TLS for S3 buckets is now possible via -<prefix>.s3.http.* and -common.storage.s3.http.* configuration options.

• Mimirtool can now verify the validity of a Mimir runtime configuration file with the mimirtool runtime-config verify command.

• Active series are now updated along with owned series. This means the number of active series for a tenant is more accurate after scaling out ingesters. As a result, ingesters now more precisely enforce tenants' series limits. This feature is only enabled when -ingester.track-ingester-owned-series or -ingester.use-ingester-owned-series-for-limits are enabled.

• Store-gateway can be configured to explicitly disable or enable tenants. This is useful in cases where compaction slows down for a tenant to temporarily exclude another tenant while compaction catches up without affecting other tenants.

• Remote read (/prometheus/api/v1/read) is becoming a first-class endpoint in Mimir with support in query stats logs and query blocking in the query-frontend. More coming in future releases.

Additionally, the following previously experimental features are now considered stable:

• Rules tenant federation via the source_tenants field in rule groups.
• Enabling recording, and alerting rules evaluation on a per-tenant basis.
• Limiting the number of tenants a federated query can query.

Important changes

In Grafana Mimir 2.13 the following behavior has changed:

• The default Docker image grafana/mimir is now based on the distroless image gcr.io/distroless/static-debian12.
  See Debugging distroless container images for more details on how to
  work with distroless images.

• Error logs in the ingester are now sampled at 10% by default. Sampled log lines contain (sampled 1/10). The cortex_discarded_samples_total metric still tracks all discarded samples.

• Anonymous usage statistics now include actual CPU usage instead of available CPU cores.

• Continuous-test is no longer a standalone binary and is now part of Mimir as its own target. The published Docker image has been updated to use the new packaging.

The following deprecated configuration options are removed in Grafana Mimir 2.13:

• The configuration option -log.buffered, which was deprecated in 2.11 and is now enabled by default.

Experimental features

Grafana Mimir 2.13 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

• Experimental support for server-side circuit breakers in ingesters on read and write requests. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options and further configured via the -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options.

The following configuration options are deprecated and will be removed in a future Grafana Mimir release:

• evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead.

Bug fixes

• OTLP ingestion: translate all HTTP 5xx errors into one of 502, 503, or 504, so that the otel-collector retries the failed requests.
• OTLP ingestion: return properly formatted protobuf error messages when ingestion fails.
• OTLP ingestion: generate target_info metric only when there are metrics in the request and there is at least one configured identifying label.
• OTLP ingestion: don't discard timeseries paired with invalid exemplars. Instead, try to ingest the timeseries and discard only the invalid exemplars.
• Subqueries: @ end() and @ start() now work correctly with queries split by time.
• Native histograms: order exemplars before ingestion to improve success rate when ingesting multiple exemplars.
• Native histograms: return HTTP 400 on invalid native histogram samples instead of HTTP 500. The metric cortex_discarded_samples_total{reason="invalid-native-histogram"} is now incremented on invalid histogram samples.

Changelog

2.13.0-rc.1

Grafana Mimir

• [CHANGE] Build: grafana/mimir docker image is now based on gcr.io/distroless/static-debian12 image. Alpine-based docker image is still available as grafana/mimir-alpine, until Mimir 2.15. #8204 #8235
• [CHANGE] Ingester: /ingester/flush endpoint is now only allowed to execute only while the ingester is in Running state. The 503 status code is returned if the endpoint is called while the ingester is not in Running state. #7486
• [CHANGE] Distributor: Include label name in err-mimir-label-value-too-long error message: #7740
• [CHANGE] Ingester: enabled 1 out 10 errors log sampling by default. All the discarded samples will still be tracked by the cortex_discarded_samples_total metric. The feature can be configured via -ingester.error-sample-rate (0 to log all errors). #7807
• [CHANGE] Query-frontend: Query results caching and experimental query blocking now utilize the PromQL string-formatted query format rather than the unvalidated query as submitted to the frontend. #7742
  • Query results caching should be more stable as all equivalent queries receive the same cache key, but there may be cache churn on first deploy with the updated format
  • Query blocking can no longer be circumvented with an equivalent query in a different format; see Configure queries to block
• [CHANGE] Query-frontend: stop using -validation.create-grace-period to clamp how far into the future a query can span. #8075
• [CHANGE] Clamp GOMAXPROCS to runtime.NumCPU. #8201
• [CHANGE] Anonymous usage statistics tracking: add CPU usage percentage tracking. #8282
• [CHANGE] Added new metric cortex_compactor_disk_out_of_space_errors_total which counts how many times a compaction failed due to the compactor being out of disk. #8237
• [CHANGE] Anonymous usage statistics tracking: report active series in addition to in-memory series. #8279
• [CHANGE] Ruler: evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead (it has the same exact meaning and behaviour). #8295
• [CHANGE] General: remove -log.buffered. The configuration option has been enabled by default and deprecated since Mimir 2.11. #8395
• [CHANGE] Ruler: promote tenant federation from experimental to stable. #8400
• [CHANGE] Ruler: promote -ruler.recording-rules-evaluation-enabled and -ruler.alerting-rules-evaluation-enabled from experimental to stable. #8400
• [CHANGE] General: promote -tenant-federation.max-tenants from experimental to stable. #8400
• [FEATURE] Continuous-test: now runable as a module with mimir -target=continuous-test. #7747
• [FEATURE] Store-gateway: Allow specific tenants to be enabled or disabled via -store-gateway.enabled-tenants or -store-gateway.disabled-tenants CLI flags or their corresponding YAML settings. #7653
• [FEATURE] New -<prefix>.s3.bucket-lookup-type flag configures lookup style type, used to access bucket in s3 compatible providers. #7684
• [FEATURE] Querier: add experimental streaming PromQL engine, enabled with -querier.promql-engine=mimir. #7693 #7898 #7899 #8023 #8058 #8096 #8121 #8197 #8230 #8247 #8270 #8276 #8277 #8291 #8303 #8340 #8256 #8348
• [FEATURE] New /ingester/unregister-on-shutdown HTTP endpoint allows dynamic access to ingesters' -ingester.ring.unregister-on-shutdown configuration. #7739
• [FEATURE] Server: added experimental PROXY protocol support. The PROXY protocol support can be enabled via -server.proxy-protocol-enabled=true. When enabled, the support is added both to HTTP and gRPC listening ports. #7698
• [FEATURE] Query-frontend, querier: new experimental /cardinality/active_native_histogram_metrics API to get active native histogram metric names with statistics about active native histogram buckets. #7982 #7986 #8008
• [FEATURE] Alertmanager: Added -alertmanager.max-silences-count and -alertmanager.max-silence-size-bytes to set limits on per tenant silences. Disabled by default. #8241 #8249
• [FEATURE] Ingester: add experimental support for the server-side circuit breakers when writing to and reading from ingesters. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options. Further -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options for configuring circuit-breaker are available. Added metrics cortex_ingester_circuit_breaker_results_total, `cortex_i...

2.13.0-rc.0

This release contains 476 PRs from 67 authors, including new contributors Anthony Keydel, Armand Grillet, AvivGuiser, Daniel R. Dagfinrud, David Collier-Brown, David Grant, Dimitris Alo, Enrique Garbi, Erik Sommer, Ian Halliday, InventiveCoder, Kevin Mingtarja, Lasse Hels, Pangidoan Butar, Quentin Bisson, Rens Groothuijsen, René Gärtner, Ross Brunson, Santiago, Seiya, Spyros Panagiotopoulos, Yuri Nikolic, Zied ABID, kahirokunn, lasermoth. Thank you!

Grafana Mimir version 2.13.0-rc.0 release notes

Features and enhancements

• Improved CPU performance in processing queries with regular expressions that contain many alternations (for example foo|bar|baz|...).

• Improved OTLP ingestion performance by using native translation, which reduces memory usage in the distributor by up to 30% and CPU usage by up to 8%.

• Configurable S3 bucket lookup type improves interoperability with S3-compatible providers, such as Tencent and Alibaba. Configure lookup type via -<prefix>.s3.bucket-lookup-type or -common.storage.s3-bucket-lookup-type.

• Configuration of TLS for S3 buckets is now possible via -<prefix>.s3.http.* and -common.storage.s3.http.* configuration options.

• Mimirtool can now verify the validity of a Mimir runtime configuration file with the mimirtool runtime-config verify command.

• Active series are now updated along with owned series. This means the number of active series for a tenant is more accurate after scaling out ingesters. As a result, ingesters now more precisely enforce tenants' series limits. This feature is only enabled when -ingester.track-ingester-owned-series or -ingester.use-ingester-owned-series-for-limits are enabled.

• Store-gateway can be configured to explicitly disable or enable tenants. This is useful in cases where compaction slows down for a tenant to temporarily exclude another tenant while compaction catches up without affecting other tenants.

• Remote read (/prometheus/api/v1/read) is becoming a first-class endpoint in Mimir with support in query stats logs and query blocking in the query-frontend. More coming in future releases.

Additionally, the following previously experimental features are now considered stable:

• Rules tenant federation via the source_tenants field in rule groups.
• Enabling recording, and alerting rules evaluation on a per-tenant basis.
• Limiting the number of tenants a federated query can query.

Important changes

In Grafana Mimir 2.13 the following behavior has changed:

• The default Docker image grafana/mimir is now based on the distroless image gcr.io/distroless/static-debian12.
  See Debugging distroless container images for more details on how to
  work with distroless images.

• Error logs in the ingester are now sampled at 10% by default. Sampled log lines contain (sampled 1/10). The cortex_discarded_samples_total metric still tracks all discarded samples.

• Anonymous usage statistics now include actual CPU usage instead of available CPU cores.

• Continuous-test is no longer a standalone binary and is now part of Mimir as its own target. The published Docker image has been updated to use the new packaging.

The following deprecated configuration options are removed in Grafana Mimir 2.13:

• The configuration option -log.buffered, which was deprecated in 2.11 and is now enabled by default.

Experimental features

Grafana Mimir 2.13 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

• Experimental support for server-side circuit breakers in ingesters on read and write requests. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options and further configured via the -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options.

The following configuration options are deprecated and will be removed in a future Grafana Mimir release:

• evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead.

Bug fixes

• OTLP ingestion: translate all HTTP 5xx errors into one of 502, 503, or 504, so that the otel-collector retries the failed requests.
• OTLP ingestion: return properly formatted protobuf error messages when ingestion fails.
• OTLP ingestion: generate target_info metric only when there are metrics in the request and there is at least one configured identifying label.
• OTLP ingestion: don't discard timeseries paired with invalid exemplars. Instead, try to ingest the timeseries and discard only the invalid exemplars.
• Subqueries: @ end() and @ start() now work correctly with queries split by time.
• Native histograms: order exemplars before ingestion to improve success rate when ingesting multiple exemplars.
• Native histograms: return HTTP 400 on invalid native histogram samples instead of HTTP 500. The metric cortex_discarded_samples_total{reason="invalid-native-histogram"} is now incremented on invalid histogram samples.

Changelog

2.13.0-rc.0

Grafana Mimir

• [CHANGE] Build: grafana/mimir docker image is now based on gcr.io/distroless/static-debian12 image. Alpine-based docker image is still available as grafana/mimir-alpine, until Mimir 2.15. #8204 #8235
• [CHANGE] Ingester: /ingester/flush endpoint is now only allowed to execute only while the ingester is in Running state. The 503 status code is returned if the endpoint is called while the ingester is not in Running state. #7486
• [CHANGE] Distributor: Include label name in err-mimir-label-value-too-long error message: #7740
• [CHANGE] Ingester: enabled 1 out 10 errors log sampling by default. All the discarded samples will still be tracked by the cortex_discarded_samples_total metric. The feature can be configured via -ingester.error-sample-rate (0 to log all errors). #7807
• [CHANGE] Query-frontend: Query results caching and experimental query blocking now utilize the PromQL string-formatted query format rather than the unvalidated query as submitted to the frontend. #7742
  • Query results caching should be more stable as all equivalent queries receive the same cache key, but there may be cache churn on first deploy with the updated format
  • Query blocking can no longer be circumvented with an equivalent query in a different format; see Configure queries to block
• [CHANGE] Query-frontend: stop using -validation.create-grace-period to clamp how far into the future a query can span. #8075
• [CHANGE] Clamp GOMAXPROCS to runtime.NumCPU. #8201
• [CHANGE] Anonymous usage statistics tracking: add CPU usage percentage tracking. #8282
• [CHANGE] Added new metric cortex_compactor_disk_out_of_space_errors_total which counts how many times a compaction failed due to the compactor being out of disk. #8237
• [CHANGE] Anonymous usage statistics tracking: report active series in addition to in-memory series. #8279
• [CHANGE] Ruler: evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead (it has the same exact meaning and behaviour). #8295
• [CHANGE] General: remove -log.buffered. The configuration option has been enabled by default and deprecated since Mimir 2.11. #8395
• [CHANGE] Ruler: promote tenant federation from experimental to stable. #8400
• [CHANGE] Ruler: promote -ruler.recording-rules-evaluation-enabled and -ruler.alerting-rules-evaluation-enabled from experimental to stable. #8400
• [CHANGE] General: promote -tenant-federation.max-tenants from experimental to stable. #8400
• [FEATURE] Continuous-test: now runable as a module with mimir -target=continuous-test. #7747
• [FEATURE] Store-gateway: Allow specific tenants to be enabled or disabled via -store-gateway.enabled-tenants or -store-gateway.disabled-tenants CLI flags or their corresponding YAML settings. #7653
• [FEATURE] New -<prefix>.s3.bucket-lookup-type flag configures lookup style type, used to access bucket in s3 compatible providers. #7684
• [FEATURE] Querier: add experimental streaming PromQL engine, enabled with -querier.promql-engine=mimir. #7693 #7898 #7899 #8023 #8058 #8096 #8121 #8197 #8230 #8247 #8270 #8276 #8277 #8291 #8303 #8340 #8256 #8348
• [FEATURE] New /ingester/unregister-on-shutdown HTTP endpoint allows dynamic access to ingesters' -ingester.ring.unregister-on-shutdown configuration. #7739
• [FEATURE] Server: added experimental PROXY protocol support. The PROXY protocol support can be enabled via -server.proxy-protocol-enabled=true. When enabled, the support is added both to HTTP and gRPC listening ports. #7698
• [FEATURE] Query-frontend, querier: new experimental /cardinality/active_native_histogram_metrics API to get active native histogram metric names with statistics about active native histogram buckets. #7982 #7986 #8008
• [FEATURE] Alertmanager: Added -alertmanager.max-silences-count and -alertmanager.max-silence-size-bytes to set limits on per tenant silences. Disabled by default. #8241 #8249
• [FEATURE] Ingester: add experimental support for the server-side circuit breakers when writing to and reading from ingesters. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options. Further -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options for configuring circuit-breaker are available. Added metrics cortex_ingester_circuit_breaker_results_total, `cortex...