keyAuth by Vault get 401 over and over again

[rongcheng-000]

I deployed my apisix in k8s with apisix-ingress-controller.Trying to make Vault as a key manager,I deployed httpbin as backend service and opened keyAuth func. Here is my apisixConsumer.yaml , and helm values.yaml setting
p.s. I built Vault on my local with Docker

apisixConsumer.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixConsumer
metadata:
  name: user1
  namespace: default
spec:
  ingressClassName: apisix
  authParameter:
    keyAuth:
      value:
        key: "$secrets://vault/1/auth-key/value"
      #  key: "123456"

Here is my apisix config.yaml

apisix
  vault:
    host: http://10.20.85.22:8200
    timeout: 10
    token: root
    prefix: "secret/apisix"
  secrets:
    - id: vault/1
      uri: "http://10.20.85.22:8200"
      token: root
      prefix: "secret/apisix"

The kv version of Vault is set to be kv=1 , but still can't looking for correct key, here is my testing cmds and responses

> curl --header "X-Vault-Token: root" http://10.20.85.22:8200/v1/secret/apisix/auth-key
{"request_id":"3abf1c26-dc03-fe08-9ed8-261bc545e1c5","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"value":"123456"},"wrap_info":null,"warnings":null,"auth":null}
> curl http://localhost:61150/ -H "apikey: 123456"
{"message":"Invalid API key in request"}

---

apisix logs:
find_consumer(): failed to find consumer: invalid api key, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", host: "localhost:61150"
run_plugin(): key-auth exits with http status code 401, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", host: "localhost:61150"

Pls help me figure out if I missed something
Btw, when I switched key in apisixConsumer to " key: 123456" , it worked , so the keyAuth func is fine

[dosubot[bot]]

Hi @rongcheng-000! I'm Dosu and I’m helping the apisix team.

Your Vault secret reference syntax in apisixConsumer.yaml is incorrect for APISIX. It should follow $secret://vault/{id}/{secret_name}/{key} where {id} matches your Vault secret resource, {secret_name} is the Vault path, and {key} is the field inside the secret data. For your setup, you should:

1. Create the Vault secret with the correct key name:

   vault kv put secret/apisix/user1 auth-key=123456

   This creates a secret at secret/apisix/user1 with key auth-key.

2. Reference it in your consumer like:

   key: "$secret://vault/user1/auth-key"

   Here, vault is your secret manager ID, user1 is the secret name, and auth-key is the key inside the Vault secret.

Your current reference ($secrets://vault/auth-key/value) does not match the expected format and will not work [docs]. APISIX expects the key name in Vault to match the last segment of the $secret:// URI [vault.lua].

Update your Vault secret and consumer reference, then retest. If you still get 401, check APISIX logs for Vault retrieval errors and confirm your consumer is applied in the cluster.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[dosubot[bot]]

The reason your secrets section is missing from config.yaml is likely due to how your Helm values.yaml is structured. The APISIX Helm chart only renders the secrets section if the values.yaml matches the chart's expected schema. If you put secrets under apisix: in values.yaml but the chart expects it at the root or under a different key, it will be ignored and not appear in config.yaml, which breaks Vault integration and causes 401 errors from key-auth [docs].

Check your Helm chart's documentation or templates to confirm where the secrets block should go. For most recent charts, it should look like this at the root level of values.yaml (not nested under apisix:):

secrets:
  - id: vault/1
    uri: "http://10.20.85.22:8200"
    token: root
    prefix: "secret/apisix"

After you update values.yaml, upgrade/redeploy with Helm and verify that config.yaml in your APISIX pod contains the secrets section. Only then will $secret://vault/1/user1/auth-key references work.

Also, make sure you use $secret:// (not $secrets://) in your ApisixConsumer, and that the id in secrets matches the one in your secret URI [reference].

Once config.yaml is correct, reload APISIX and your key-auth with Vault should work.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[rongcheng-000]

It still didn't work....

curl http://localhost:58942/ -H "apikey: 123456"
{"message":"Invalid API key in request"}
curl --header "X-Vault-Token: root" http://10.20.85.22:8200/v1/secret/apisix/user1
{"request_id":"a8294bc8-2d31-ffb5-9b1d-9c13757fde18","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"auth-key":"123456"},"wrap_info":null,"warnings":null,"auth":null}

I could get data with API of Vault , but just couldn't get key value through APISIX

    keyAuth:
      value:
        key: "$secret://vault/1/user1/auth-key"

secrets:
  - id: vault/1
    uri: "http://10.20.85.22:8200"
    token: root
    prefix: "secret/apisix" 

I really can't figure out why...

[dosubot[bot]]

The most likely reason APISIX key-auth with Vault is still returning 401 is that your running APISIX config.yaml does not actually contain the secrets section—so APISIX cannot fetch secrets from Vault, even though your values.yaml looks correct. The Helm chart only renders the secrets section if it’s placed at the root of values.yaml, not under apisix:. You need to:

1. Exec into your APISIX pod and inspect /usr/local/apisix/conf/config.yaml (or wherever your config is mounted) to confirm the secrets section is present and matches your $secret:// URI exactly (id, uri, token, prefix) docs.
2. Make sure your values.yaml has secrets: at the root, not nested under apisix:, like:

   secrets:
     - id: vault/1
       uri: "http://10.20.85.22:8200"
       token: root
       prefix: "secret/apisix"

3. Redeploy with Helm and verify config.yaml in the pod contains the secrets section.
4. Confirm your ApisixConsumer uses $secret://vault/1/user1/auth-key and that the Vault secret at secret/apisix/user1 contains auth-key: 123456.

If the secrets section is missing or the id/URI segments don’t match, APISIX will silently fail to fetch the secret and always return 401 reference. Double-check for typos and segment mismatches. Once config.yaml is correct, reload APISIX and your Vault key-auth should work.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[rongcheng-000]

It still can't work...
Does APISIX have any other ways to store keys ? except GCP or AWS

[dosubot[bot]]

APISIX officially supports storing keys for authentication plugins like key-auth in HashiCorp Vault, AWS Secrets Manager, and GCP Secret Manager. There’s no support in the open-source version for other backends such as Kubernetes Secrets, Azure Key Vault, Consul, etcd, or local file-based storage for dynamic secret retrieval via the $secret:// reference. If you need more options, API7 Enterprise (commercial APISIX) allows storing secrets directly in its own encrypted storage, but this isn’t available in open-source APISIX. See the docs for details: APISIX secret management.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[Baoyuantop]

Hi @rongcheng-000, you can refer to https://docs.api7.ai/apisix/how-to-guide/security/secrets-management/manage-secrets-in-hashicorp-vault/

If the problem persists, please submit an issue with a detailed description of your problem.

[rongcheng-000]

It still can't work....
I just followed the docs above, but still got logs from apisix pods like:

find_consumer(): failed to find consumer: invalid api key
run_plugin(): key-auth exits with http status code 401

[rongcheng-000]

Does it really work that APISIX CRD - ApisixConsumer can send secret_uri request to Vault for getting keys ?
If someone really did it, could you please show me how~~

[Baoyuantop]

After reviewing the relevant code, it was found that the ApisixConsumer CRD currently only supports Kubernetes secret references and does not support vault references.

[rongcheng-000]

Could you please show me the docs you found?
Anyway, thanks...
It really has troubled me a lot for long time

[Baoyuantop]

https://apisix.apache.org/docs/ingress-controller/reference/apisix-ingress-controller/api-reference/#apisixconsumerkeyauth

[ikromrr]

I deployed my apisix in k8s with apisix-ingress-controller.Trying to make Vault as a key manager,I deployed httpbin as backend service and opened keyAuth func. Here is my apisixConsumer.yaml , and helm values.yaml setting

p.s. I built Vault on my local with Docker

apisixConsumer.yaml

apiVersion: apisix.apache.org/v2

kind: ApisixConsumer

metadata:

  name: user1

  namespace: default

spec:

  ingressClassName: apisix

  authParameter:

    keyAuth:

      value:

        key: "$secrets://vault/1/auth-key/value"

      #  key: "123456"

Here is my apisix config.yaml

apisix

  vault:

    host: http://10.20.85.22:8200

    timeout: 10

    token: root

    prefix: "secret/apisix"

  secrets:

    - id: vault/1

      uri: "http://10.20.85.22:8200"

      token: root

      prefix: "secret/apisix"

The kv version of Vault is set to be kv=1 , but still can't looking for correct key, here is my testing cmds and responses

> curl --header "X-Vault-Token: root" http://10.20.85.22:8200/v1/secret/apisix/auth-key

{"request_id":"3abf1c26-dc03-fe08-9ed8-261bc545e1c5","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"value":"123456"},"wrap_info":null,"warnings":null,"auth":null}

> curl http://localhost:61150/ -H "apikey: 123456"

{"message":"Invalid API key in request"}

---

apisix logs:

find_consumer(): failed to find consumer: invalid api key, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", host: "localhost:61150"

run_plugin(): key-auth exits with http status code 401, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", host: "localhost:61150"

Pls help me figure out if I missed something

Btw, when I switched key in apisixConsumer to " key: 123456" , it worked , so the keyAuth func is fine

I found this issue also in the apisix monolithic, I have been trying to configure the secret manager using vault like in the documentation but also got 401 like yours.

but while consuming an api using the key like $secret://vault/1/jack/key-auth its got 200. Meaning apisix doesn't look up the value as a reference but as a key