Buffering per service without Kafka

[brycefisher]

I'm interested in whether vector might be a good fit for my use case -- I have an existing solution, but it doesn't seem to solve a specific problem I have. I have a kubernetes cluster using the prometheus operator to scrape metrics from many different workloads. Unfortunately, if a single Deployment or other workload begins to create many pods rapidly, this can produce too many series for my metrics sink.

In my current solution, I don't have a good way to proactively limit the number of cumulative series from a single ServiceMonitor (all pods from a Deployment, for example). And no single pod produces so many metrics that I can use prometheus scrape_config.label_limits.

If I used vector without kafka as described in the docs, would it be possible to have vector drop metrics from a single ServiceMonitor based on the cumulative cardinality of that service monitor?

[jszwedko]

I think that might be possible using the throttle transform. That transform can be used to limit the number of events per time window for a given key.

[pront]

Vector does not currently have a built-in limit on the total number of distinct series per ServiceMonitor.

The throttle transform suggested above limits log event rates, so it would not enforce this. tag_cardinality_limit limits distinct values for each tag. For example, two pod values combined with two status values can still produce four series.

#12996 tracks limiting metric cardinality, and #21112 tracks grouping limits by a key. Your per-ServiceMonitor use case would be useful context on those requests.