Alertmanager secrets

[OfficerKoo]

How are we supposed to deploy Alertmanager secrets(OpsGenie API keys for example), if they are stored in plaintext in config file.
Vanilla Alertmanager allows you to store secrets in a file, but this is disabled in Mimir Alertmanager API, and there is no other way to upload tenant configurations.

Is there some best practise to do this?
I really don't want to add some templating engine to a pipeline, or inject header with reverse proxy.

DEBU[0001] checking response                             status="400 Bad Request"
ERRO[0001] response                                      body="error validating Alertmanager config: setting smtp_auth_password_file, password_file, bearer_token_file, auth_password_file or credentials_file is not allowed\n" status="400 Bad Request"
mimirtool: error: POST request to https://mimir.internal.infra.loveos.io/api/v1/alerts failed: server returned HTTP status: 400 Bad Request, body: "error validating Alertmanager config: setting smtp_auth_password_file, password_file, bearer_token_file, auth_password_file or credentials_file is not allowed\n", try --help

[dimitarvdimitrov]

You can use environment variables inside the configuration.

"Injecting credentials" shows how to do it for the Helm chart.

And "Use environment variables in the configuration" is a general paragraph about how environment variables in the config work.

[lblazewski]

Hey, I would like to bump this since I have stumbled upon this issue recently. I think this is a serious limitation and secrets support should be added to the alertmanager config parsing.

[lukasmrtvy]

@lblazewski there's no issue created afaik, maybe this would be a good start

[taobojlen]

I created a feature request here: #8259

[mac133k]

I would really like to have the feature in mimirtool to expand env vars placed in tenant's alertmanager config. If the config is under revision control the settings that are env-specific or are sensitive data could be filled using env var when mimirtool is dispatched as a K8s job to load alertmanager configuration for a specific tenant in a specific staging env to Mimir Alertmaanger. Currently this is only possible using external tools or scripts that need to wrap around mimirtool or run in an init container.

[marshallford]

Has anyone solved this or found a workaround?