Metric events - what is the algorithm of summary metrics?

[AlexSTAL1980]

Hi!

Quote from the page Metric events:
Similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

What is the algorithm for "sliding time window"?
By a certain number of metrics? Or for a certain time period? Or some other algorithm?

I tried to analyze the source code and found the "entry point" in file \vector-master\src\sinks\prometheus\collector.rs

                MetricValue::Distribution {
                    samples,
                    statistic: StatisticKind::Summary,
                } => {
                    if let Some(statistic) = DistributionStatistic::from_samples(samples, quantiles)

where quantiles are calculated. But I don't have enough experience to understand how and where the "samples" array is formed.

Please tell me the complete sampling algorithm.

[jszwedko]

Hey! When converting a Prometheus distribution into an internal Vector "Summary" metric, there are no samples. There are just the summary statistics: count, sum, min, max, and quantiles. See https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#histograms-and-summaries for details on how Prometheus serializes histograms.

[AlexSTAL1980]

Hey! Did I understand correctly from the answer that the above-mentioned documentation Vector "Metric events" contains an error, and there is no "sliding time window", and the statistics are calculated for the entire available period?

If I understood correctly, then in my opinion this is incorrect, and quantiles for export to Prometheus must be calculated in the vector for a configurable time interval or for a specified number of values.

I will provide a translation of an excerpt from the article:

Usually for metrics we sample data, that is, from the entire set we select only some elements in order to calculate percentiles and other properties from them. How to select a finite number of elements from an infinite series of data? You will have to sacrifice something, there are different algorithms for this. We store, for example, up to 100 elements, and when we need to write a new one, we decide which element to throw out:

• Sliding Window: just throw out the oldest element. Very easy to implement and understand
• Forward Decay: throw away a random element, but the older it is, the more likely it is to be thrown away. The probability can grow in any way, for example, exponentially. The idea is that old values ​​still slightly affect the properties of the array, so this sample describes our infinite series of data a little more accurately.

Otherwise, it turns out that Max always increases (and never decreases), Min always decreases (and never increases), and the quantiles themselves are always averaged over the entire available range.

And one of the purposes of this type of metrics is a visual representation of the query execution time, which should change quickly (for example, over the last 5 minutes, but not for the entire available history).

[jszwedko]

Summary statistics do have a timestamp. That's a good question about whether they are expected to represent the entire period or just a window. Metrics can be either "incremental" or "absolute" so perhaps an "incremental summary" would just represent the time window since the last summary metric.

For Prometheus, though, I believe the summaries are absolute (that is: they represent a summary of the entire range).

[AlexSTAL1980]

Please tell me, why do we need to know the "average temperature in the hospital"?
I found a good technical article: How summary metrics work in Prometheus.

Main thoughts:

• Prometheus has its own client, in the example in Java:
  Summary metrics are implemented in the Prometheus client libraries, like client_golang or client_java. You call the library in your REST application, and the library will calculate the percentiles for you.

• a sliding time window is set in the client:
  You don’t want to get latency metrics for the entire runtime of an application. You want to look at a reasonable time interval. For example, client_java implements a configurable sliding window which is 10 minutes by default.

That is, when calculating the summary metrics, there must be one more parameter - a sliding time window, as it is implemented in the same Prmetheus client.

Otherwise it doesn't make any practical sense.

[jszwedko]

I feel like I've lost what you are trying to do here 😄 Can you describe your use-case?

[AlexSTAL1980]

Unfortunately, I am not familiar with the Rust programming language and cannot understand in the source code where and by what algorithm the summary metric is calculated. According to the links I provided above, there are at least 3 algorithms for calculation.

Most likely, for a full implementation of the summary metrics, a separate transformer will be required, which will have a "sliding_time_window" parameter, specified in seconds or minutes, where the calculation will take place (now there is a similar "Aggregate", which has a time window for calculating "interval_ms").

That is, it is necessary to implement a full calculation of the summary metrics in Vector

The current implementation is wrong

[AlexSTAL1980]

Addition
And the usage scenario itself is simple - monitoring the duration of requests, response sizes, etc.

[jszwedko]

I see, yes, I think what you might be asking for is the ability to calculate summary metrics.

[AlexSTAL1980]

Unfortunately, the language barrier prevents me from understanding your answer accurately.

[KimMJ]

hello. I too use log-to-metric and I believe the histogram needs a sliding time window. I wanted to get a latency histogram for a specific api from nginx logs, which can be implemented via bucket, but also via summary type. I want to use the summary type to reduce the load on the metrics system, and simply see the p95 stats for each pod. So I initially tried to use the summary type in vector, but after realizing that it gives stats for the entire time period, I ended up using the method via bucket, which is semantically more correct, but also has the problem of creating too many series in some cases. Perhaps people who use the summary type would like to get stats over a period of time, and a feature toggle would be a good way to give them a choice.

[pront]

For log_to_metric → prometheus_exporter, there is no sliding observation window in Vector 0.60.0. An active series keeps accumulating observations.

With distributions_as_summaries: true, the path is:

1. log_to_metric emits one distribution sample for each valid input value, with a count of 1.
2. The exporter converts those observations into a DDSketch, which stores approximate quantiles, and merges updates for the same metric name and labels.
3. Scrapes read the accumulated sketch. They do not reset it.
4. flush_period_secs expires inactive series as a whole. It does not remove old observations from a series that keeps receiving updates.

The from_samples function you found calculates statistics from the supplied samples; it does not select a time window. The current exporter uses the sketch conversion and accumulation path.

I verified this locally with flush_period_secs: 2: after one observation of 100 followed by ten observations of 1 over five seconds, the exporter still reported count 11, sum 110, and p95 approximately 100.

For latency percentiles over the last five minutes, histograms queried over a time range are an option today. Please open a feature request for a configurable sliding window for summaries.