Mimir-ruler destination_tenant? Multi tenancy?

[RobertSalong]

We have a mutli-tenant system for mimir:

One (main tenant) for active metrics with 30 days retention
One (LTS tenant) where we aggregate/downsample and have a retention for 2 years.
For metric trends to see how performance has evolved over a longer period etc.

For background we are running all Grafana stacks/components OSS in Amazon EKS.

We do most of our rule recording and routing in federated prometheus. And this has worked well so far. Nowdays we also have some data in Mimir main tenant that we would like to downsample and send to the LTS tenant.

Now we could to this by setting up a ruler, configuring it to the LTS tenant,

volumeMounts:
  - name: mimir-lts-rules
     mountPath: "/tmp/rules/<lts_tenant>"

enable tenant federation.

tenant_federation:
  enabled: true

and set source tenant on our rule group(s) to the main one.

groups:
  - name: <group_name>
     source_tenants: ["<main_tenant>"]

and this works great. No issues... in theory.

But now we have a problem where we need to run some rules for LTS tenant and some rules for Main tenant. And i for the life in me cannot figure out a way to do this properly.

Is there a proper way to have two sets of recording rules:
One set for main tenant, that takes data from itself and saves the recorded metrics into main tenant
One set for LTS tenant, that takes data from main tenant aggregates and downsamples and saves the recorded metrics into LTS tenant?

I tried about various different methods to do this and found one that seems very incorrect but as a workaround it is currently functional.

We have two separate manifests for ruler:

mimir_ruler.yml
runs in main tenant

volumeMounts:
  - name: mimir-rules
     mountPath: "/tmp/rules/<main_tenant>"

Does not use source_tenants option.

mimir_ruler_lts.yml
runs in lts tenant but gets data from main tenant

volumeMounts:
  - name: mimir-lts-rules
     mountPath: "/tmp/rules/<lts_tenant>"

Uses source_tenants: ["<main_tenant>"]

Now initally they tried joining gossip ring and sync the rules but this made both go crazy and either one or none would work. Then i found that you can kind of isolate the rulers completely so they don't ever know of each others existence

ruler:
  ring:
    kvstore:
      store: inmemory

And this kind of works the way we need it.... for now. I have two separate tenants and rules are being recorded ending up in the correct tenant with the correct data.

Now that i have finished writing this I'm not even sure anymore why i started. I think I'm worried that at some point with updates this configuration will break and i don't dare to lean onto this solution too heavily. Also i can't seem to get information from the lts_tenant ruler endpoints. Any request i send to ruler endpoints <mimir_url>/<endpoint> with or without header responds with data from main tenant ruler, cant seem to either correctly pass x-scope-orgid or ruler just doesn't care for it.

TL;DR: Is there a a correct way, if there is a way at all to have mimir ruler(s?) record rules in a way that one set of rules end up in one tenant and another set in the other? Everything i have read and researched points to not yet, but Is something like this in the roadmap, even considered?

[deniszh]

We successfully running multi tenant ruler in our setup. As far as I understand you can't configure destinantion tenant, it always recording result to tenant which rules created for. You can use source_tenants parameter in rule groups fo configure source tenant if needed if federation is enabled.

IMO for proper setup you'll need to

1. set it up single ruler using object storage (e.g. S3)
2. use ruler API (directly or through mimirtool) to upload rules separately for each tenant. API accepting x-scope-orgid for tenant id, mimirtool using MIMIR_TENANT_ID variable for same purpose.