Configured password (boundary/config) gets lost after a period of time?

[seanamos]

I am testing the boundary plugin in a Vault cluster (2 standbys). After +- 2 weeks, the vault plugin will stop being able to generate worker credentials, failing with authentication errors to HCP boundary. This causes the workers to crash/fail to start up.

The boundary vault mount is configured with terraform:

resource "vault_generic_endpoint" "boundary_config" {
  path                 = "${vault_mount.boundary.path}/config"
  ignore_absent_fields = true

  data_json = jsonencode({
    addr           = var.address
    login_name     = boundary_account_password.vault.login_name
    password       = boundary_account_password.vault.password
    auth_method_id = var.auth_method_id
  })
}

The credentials are static, they do not rotate. Inspecting the config with vault read boundary/config, the visible values are correct. Password cannot be confirmed as it will not be returned.

Running terraform -replace vault_generic_endpoint.boundary_config will immediately fix the issue.

I'm at a bit of a loss as to what the problem might be. It seems extremely strange that the password by itself would just go missing/get corrupt some way in Vault, but I'm not familiar with the specifics of Vault plugin development.

[seanamos]

Just to confirm, the timeframe when this happens is not +- 2 weeks. It's 7 days exactly.

[seanamos]

@jimlambrt I see you beat me to this. I hadn't looked at this in a while, but it is definitely related to the boundary client and authentication token being held indefinitely by the plugin, even after the authentication token expires.