Rate chart drop in Thanos Query

[raz08]

We are using Thanos Query in our architecture to serve metrics queries for applications and clients. Our Prometheus setup consists of 2 shards with 2 replicas each, and Thanos Query is used in front to aggregate and deduplicate data across all Prometheus instances.

We observed an issue where the rate graph shows sudden narrow dips only when queried through Thanos Query. The same query, when executed directly on Prometheus, does not show these drops — the graph appears consistent and stable.

There are no signs of data gaps or issues on the Prometheus side or in the application pipeline. This behavior appears specific to Thanos deduplication or query execution.

Attached below are screenshots comparing Thanos Query and Prometheus outputs for the same expression for reference.

Thanos, Prometheus and Golang version used:
Thanos - v0.34.1
Prometheus - 3.4.2

Thanos UI

Prometheus UI

Anything else we need to know:

Sample data which has drop

prometheusratechartdata.txt
thanosratechartdata.txt

[Chr0my]

👋 I see your are aggregating by (pod, interfaceName) on Prometheus while you add an extra shard on Thanos, so you are comparing 2 different expressions.
Also ensure you are using same resolution on each to ease the comparison