End certificates are non-compliant (Missing Extended Key Usage)

[iaftab-alam]

Hey colleagues,

Context (current behavior):
Several leaf certificates in CF-D manifest are missing Extended Key Usage properties. As of now, at least 15 leaf certificates missing EKU configurations(please see list below), which violates best practices for X.509 certificate usage and could cause TLS validation issues in strict security environments(in our case trigger this finding) and lacking least‑privilege usage signaling.

Past related discussions and fixes:
We have seen in the past, if EKU values for leaf certificates are not provided one can face the same unexpected behavior where wrong EKU values are created depending how the certificates are generated, see here, here and Slack thread (internal) for more context. Quoted behavior from earlier incident:

When the certificate is generated with BOSH interpolate, the extended_key_usage option was automatically added with TLS Web Server Authentication value only which is wrong in this case, as the certificate is used for client authentication. Therefore, it cannot be used for mutual TLS authentication between the cloud controller and log-cache. When one uses CredHub for managing the certificates, the X509v3 Extended Key Usage SSL extension is not configured at all an the certificate can be used without any problems.

Affected Leaf Certificates (Missing EKU, confirming counts—list based on current manifest snapshot from main manifest)
1. blobstore_tls 
2. blobstore_public 
3. credhub_tls 
4. mysql_server_certificate 
5. uaa_login_saml 
6. uaa_jwt_signing_cert 
7. router_ssl 
8. uaa_ssl
9. cc_public_tls 
10. logcache_ssl 
11. log_cache_proxy_tls 
12. gorouter_lb_health_tls 
13. tcp_router_lb_health_tls
14. loggregator_rlp_gateway_tls 
15. loggregator_trafficcontroller_tls 

Additional Leaf Certificates that are also missing EKU:

1. saml_oidc-key1, uaa-oidc-cert1, uaa-oidc_tls in operations/test/add-oidc-provider.yml
2. ldap_server_ssl in operations/test/enable-nfs-test-ldapserver.yml
3. haproxy_ssl in operations/use-haproxy.yml
4. metricstore_ssl, metric_store_proxy_tls, metric_store_metrics_server, metric_store_metrics_client in operations/use-metric-store.yml

One might argue that our CF-D works normally fine in our environments without specifying these EKU values and is true. For instance our community pipelines also works fine without it. Which means having the wrong EKU actively prevents intended usage in stricter environments where as missing EKU is ambiguous, but doesn’t block usage; it’s still a best-practice gap (least privilege signaling missing), yet functionally more forgiving but for stricter environments it can cause problems.

As far as I know, there are 4 different ways, how the certificates (and other config-server variables) are generated and managed within the BOSH ecosystem:

#	Method	Config Server Source	Description	Typical Use Case
1	BOSH CLI (vars-store)	None (local file)	Certificates and credentials are generated locally by the BOSH CLI using a --vars-store YAML file during deploy.	Simple, standalone deployments or dev/test CF.
2	Bootstrap BOSH	Internal, preconfigured	During bootstrap BOSH creation (via bosh create-env), BOSH itself generates the initial CA and certificates. These become the seed for later environments.	Setting up your first BOSH Director (before CredHub).
3	BOSH with External CredHub	External Config Server	Certificates and variables are generated by CredHub (either co-located or external). CredHub becomes the “source of truth” for all vars.	Production CF environments (standard with cf-deployment).
4	Calling API (CredHub API or External PKI API)	External (Custom)	Certificates are created externally by calling an API — e.g., CredHub API (/api/v1/certificates), HashiCorp Vault, or an enterprise PKI.	Advanced or enterprise setups with compliance needs.

Key differences: Internal generator default vs CredHub pass‑through. They are not feature aligned, for example, If we use bosh interpolate to generate certificate we automatically gets server_auth as EKU value. In addition to this, we have brand new PR is open on CredHub to resolve the key_usage to be added automatically for Root CAs which were missing.

Reproducing (Default EKU vs No EKU)

name: demo
variables:
- name: demo_root_ca
  type: certificate
  options:
    is_ca: true
    common_name: demo-root-ca
- name: demo_cert
  type: certificate
  options:
    ca: demo_root_ca
    common_name: demo.example.internal

# bosh interpolate mini-manifest.yml --vars-store vars.yml > /dev/null

yq '.demo_cert.certificate' vars.yml > bosh.pem | openssl x509 -in bosh.pem -text | grep -C 5 "Extended Key Usage"
                    d3:65:1e:25:3c:f5:d5:ab:76:7b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                1F:D1:9D:84:DA:8F:03:27:E1:6F:5F:FF:B9:D8:3A:21:C3:BD:DF:ED

When such certs are created with CredHub acting as config_server we didn't get this EKU extension? Is the X509v3 Extended Key Usage: with TLS Web Server Authentication extension is added by(the internal generator adds it if you didn’t specify any?) calling this part, from config_server.

Root CA EKU Issue:
The uaa-oidc_ca (in add-oidc-provider.yml includes an EKU value. While RFC 5280 permits EKU on CA certificates, including application-specific usages (e.g., server_auth, client_auth) on a CA that signs different leaf usages is typically discouraged and may be unnecessary.

Recommended Actions

In scope:

• Add explicit extended_key_usage arrays to all listed leaf certificate variables.
• Remove EKU from CA variables unless intentionally restrictive.
• Cross check and confirm if the affected CAs has correct EKU with their respective components and won't cause any down time.
  If this worth consideration, I can follow up with a PR however I have crated a branch whose status is WIP.

out of scope:

• Align generation semantics across tooling—drop the silent default in internal generator
• Update documentation/ provide audited documentation
• Involve Bosh colleagues to resolve his feature misalignment between the config_servers.