Alertmanager & Ruler: Parse configuration from storage backend

[pkarakal]

In order to create or update a new AlertManager configuration or update the recording/alerting rules, one has to use mimirtool or perform an http request to Mimir. This means that if we want to do that using CI, we need to have an ingress for Mimir that is accessible by the runner. If the CI runners are SaaS ones, one has to make the ingress internet facing and deal with authentication and firewalls/security groups. If the runners are self-hosted, then one has to deal with VPC peering.

Since both Ruler and AlertManager during each evaluation cycle load the configuration from their store, what if they could also parse yaml files containing the configuration of each component?
Basically what I was thinking is the following:

• a dev makes some changes to alertmanager and ruler and pushes it to version control
• CI runner validates and checks the files
• CI pushes the configuration to the store under the predefined path
• both ruler and alertmanager on each evaluation cycle check that directory as well
  • if the configuration files are not already loaded and in their state, they should parse the file and create the configuration on the store (basically creating the protobuf objects) and add the processed files to an internal cache that contains a mapping between the file path on the store and the digest, to avoid processing it again if the content hasn't changed
  • if the configuration file is already loaded and in their state, it should skip those.
  • optionally also delete the yaml files after their are added to the state and saved as a protobuf file

This way, alertmanager and its ingress can remain private and any change including silencing, will be done using a declerative GitOps approach. I am opening this as a draft proposal so that we can discuss this.