Replace `count()` queries in `promql/series` with the Series Metadata API

[wbollock]

promql/series validates metric existence by running bare count() queries. On high-cardinality metrics this results in long queries that can timeout and are inefficient.

Example

For example, if the rule involves a traditionally high cardinality time series like node_cpu_seconds_total, it will run:

1. Instant query with labels: count(node_cpu_seconds_total{mode="idle",job="node"})
2. If step 1 returns nothing it goes to a bare range query of count(node_cpu_seconds_total) with all label matchers stripped which can be extremely expensive.

Step 2 is the biggest problem!

pint/internal/checks/promql_series.go

Lines 247 to 300 in 2a2e9d5

	continue
	}
	labelNames = append(labelNames, lm.Name)
	}
	// 1. If foo{bar, baz} is there -> GOOD
	slog.LogAttrs(ctx, slog.LevelDebug, "Checking if selector returns anything", slog.String("check", c.Reporter()), slog.String("selector", s))
	count, err := c.instantSeriesCount(ctx, wrapExpr(s, "count"))
	if err != nil {
	problems = append(problems, problemFromError(err, entry.Rule, c.Reporter(), c.prom.Name(), Bug))
	continue
	}
	if count > 0 {
	slog.LogAttrs(ctx, slog.LevelDebug, "Found series, skipping further checks", slog.String("check", c.Reporter()), slog.String("selector", s))
	continue
	}
	promUptime, err := c.prom.RangeQuery(ctx, wrapExpr(c.prom.UptimeMetric(), "count"), params)
	if err != nil {
	slog.LogAttrs(ctx, slog.LevelWarn, "Cannot detect Prometheus uptime gaps", slog.Any("err", err), slog.String("name", c.prom.Name()))
	}
	if promUptime != nil && len(promUptime.Series.Ranges) == 0 {
	slog.LogAttrs(
	ctx, slog.LevelWarn,
	"No results for Prometheus uptime metric, you might have set uptime config option to a missing metric, please check your config",
	slog.String("name", c.prom.Name()),
	slog.String("metric", c.prom.UptimeMetric()),
	)
	}
	if promUptime == nil || len(promUptime.Series.Ranges) == 0 {
	slog.LogAttrs(
	ctx, slog.LevelWarn,
	"Using dummy Prometheus uptime metric results with no gaps",
	slog.String("name", c.prom.Name()),
	slog.String("metric", c.prom.UptimeMetric()),
	)
	promUptime = &promapi.RangeQueryResult{ // nolint: exhaustruct
	URI: c.prom.URI(),
	Series: promapi.SeriesTimeRanges{
	From: params.Start(),
	Until: params.End(),
	Step: params.Step(),
	Ranges: promapi.MetricTimeRanges{
	{
	Fingerprint: 0,
	Labels: labels.Labels{},
	Start: params.Start(),
	End: params.End(),
	},
	},
	Gaps: nil,
	},
	}
	}

Proposed Change

Replace both queries with a single call to the /api/v1/series metadata endpoint which is pretty fast:

# labeled time series existence check
GET /api/v1/series?match[]=node_cpu_seconds_total{mode="idle",job="node"}&limit=1

# historical bare check (replaces range query)
GET /api/v1/series?match[]=node_cpu_seconds_total&limit=1&start=<now-7d>&end=<now>

limit=1 is also important as we only need the first match for this check, adding some extra efficiency gains. We verified on a large production instance:

# hangs with limit
curl '.../api/v1/series?match[]=node_cpu_seconds_total'

# returns immediately and stops on the first match
curl '.../api/v1/series?match[]=node_cpu_seconds_total&limit=1'

This should be a general improvement to the check with no downsides I can think of. As usual happy to contribute the change myself if you would like!

[wbollock]

hi @prymitive wondering if you have thoughts on this one? wanted to get your approval before sending a PR