Service Account Token for Image Pulls doesn't work

[jicowan]

What happened:
I enabled the KubeletServiceAccountTokenForCredentialProviders feature gate on the kubelet and created the following CredentialProviderConfig:

{
  "apiVersion": "kubelet.config.k8s.io/v1",
  "kind": "CredentialProviderConfig",
  "providers": [
    {
      "name": "ecr-credential-provider",
      "matchImages": [
        "${ACCOUNT}.dkr.ecr.*.amazonaws.com",
        "${ACCOUNT}.dkr.ecr-*.*.amazonaws.com"
      ],
      "defaultCacheDuration": "12h",
      "apiVersion": "credentialprovider.kubelet.k8s.io/v1",
      "args": ["get-credentials"],
      "env": [
        {
          "name": "AWS_REGION",
          "value": "${REGION}"
        }
      ],
      "tokenAttributes": {
        "serviceAccountTokenAudience": "sts.amazonaws.com",
        "cacheType": "Token",
        "requireServiceAccount": false,
        "optionalServiceAccountAnnotationKeys": ["eks.amazonaws.com/role-arn"]
      }
    }
  ]
}

With this configuration, the provider appears to look for the project token in ALL pods instead on the pods from my private registry. I couldn't find a way to tell the provider to use the EC2 instance profile to pull the images (I believe it should do that when no token is found, i.e. when the pod is does not have a service account that is mapped to an IAM role using IRSA). The following appears repeatedly in the kubelet logs:

Oct 14 16:42:44 ip-192-168-40-236.us-west-2.compute.internal kubelet[3037]: E1014 16:42:44.210176    3037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"aws-vpc-cni-init\" with ImagePullBackOff: \"Back-off pulling image \\\"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init:v1.20.1-eksbuild.3\\\": ErrImagePull: failed to pull and unpack image \\\"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init:v1.20.1-eksbuild.3\\\": failed to resolve reference \\\"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init:v1.20.1-eksbuild.3\\\": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials\"" pod="kube-system/aws-node-f7sf8" podUID="226e4610-501f-485d-a173-b96d596e5285"
Oct 14 16:42:47 ip-192-168-40-236.us-west-2.compute.internal kubelet[3037]: E1014 16:42:47.645317    3037 kubelet.go:3021] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

If I try running the ecr-credential-provider manually with a pod that uses IRSA, I am able to get a ECR login.

What you expected to happen:
I expect the provider to fall back to the instance profile of the node if the token is not present.

Anything else we need to know?:
Here's the userdata I'm using:

#!/bin/bash
set -ex

# Disable the automatic nodeadm service since we'll run it manually
systemctl disable nodeadm-config.service || true

# Get cluster details from EC2 tags and EKS API
REGION=$(ec2-metadata --availability-zone | sed 's/placement: \(.*\).$/\1/')
INSTANCE_ID=$(ec2-metadata --instance-id | cut -d " " -f 2)
CLUSTER_NAME=$(aws ec2 describe-tags --region ${REGION} --filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=eks:cluster-name" --query 'Tags[0].Value' --output text)

# Get cluster details from EKS
API_SERVER=$(aws eks describe-cluster --region ${REGION} --name ${CLUSTER_NAME} --query 'cluster.endpoint' --output text)
CA_CERT=$(aws eks describe-cluster --region ${REGION} --name ${CLUSTER_NAME} --query 'cluster.certificateAuthority.data' --output text)
CIDR=$(aws eks describe-cluster --region ${REGION} --name ${CLUSTER_NAME} --query 'cluster.kubernetesNetworkConfig.serviceIpv4Cidr' --output text)
DNS_IP=$(echo ${CIDR} | awk -F'.' '{print $1"."$2"."$3".10"}')

# Generate NodeConfig and run nodeadm
cat > /tmp/nodeadm-config.yaml <<EOF
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: ${CLUSTER_NAME}
    apiServerEndpoint: ${API_SERVER}
    certificateAuthority: ${CA_CERT}
    cidr: ${CIDR}
  kubelet:
    config:
      clusterDNS:
      - ${DNS_IP}
      featureGates:
        ContainerCheckpoint: true
        KubeletServiceAccountTokenForCredentialProviders: true
EOF

# Run nodeadm with the generated config
# nodeadm will generate the default credential provider config
/usr/bin/nodeadm init --config-source file:///tmp/nodeadm-config.yaml

# Now overwrite the ECR credential provider config to include tokenAttributes
# This happens AFTER nodeadm has finished initialization
cat > /etc/eks/image-credential-provider/config.json <<EOF
{
  "apiVersion": "kubelet.config.k8s.io/v1",
  "kind": "CredentialProviderConfig",
  "providers": [
    {
      "name": "ecr-credential-provider",
      "matchImages": [
        "820537372947.dkr.ecr.*.amazonaws.com",
        "820537372947.dkr.ecr-*.*.amazonaws.com"
      ],
      "defaultCacheDuration": "12h",
      "apiVersion": "credentialprovider.kubelet.k8s.io/v1",
      "args": ["get-credentials"],
      "env": [
        {
          "name": "AWS_REGION",
          "value": "${REGION}"
        }
      ],
      "tokenAttributes": {
        "serviceAccountTokenAudience": "sts.amazonaws.com",
        "cacheType": "Token",
        "requireServiceAccount": false,
        "optionalServiceAccountAnnotationKeys": ["eks.amazonaws.com/role-arn"]
      }
    }
  ]
}
EOF

# Restart kubelet to pick up the new credential provider config
systemctl restart kubelet

Environment:

• Kubernetes version (use kubectl version): 1.34
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): EKS Optimized AMI with custom launch template
• Kernel (e.g. uname -a):
• Install tools:
• Others:

/kind bug

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jicowan]

If I add all ECR registries to the matchImages section, the kubelet will invoke the ECR credential provider for every matching image pull.
And, as currently implemented, the provider will:

Always attempt to use a service account token if one is present (even if it’s a standard Kubernetes token and not an IRSA token).
Fail to distinguish between IRSA tokens (usable for AWS STS AssumeRoleWithWebIdentity) and generic tokens.
Not fall back to the instance profile if the token is present but not valid for IRSA.

This means that:
For pods with a projected service account token that is not an IRSA token, image pulls will fail, even if the node’s instance profile has permissions.
For pods without a service account token, it will use the instance profile (and succeed if the node is configured correctly).
For pods with a valid IRSA token, it will succeed using IRSA.

The fix needs to distinguish token type

To safely use the credential provider for all registries, it needs to:
Detect if the token is an IRSA token (i.e., associated with an IAM role via annotation AND can be used for web identity federation).
Ignore non-IRSA tokens and fall back to the instance profile.

This could be accomplished by:
Checking for the presence of the eks.amazonaws.com/role-arn annotation on the service account.
Attempting to validate the projected token (structure, claims, etc.) before calling AWS STS.
If the above checks fail, skip the IRSA flow and use the default chain (instance profile).

[jicowan]

The issue is specifically in the mismatch between the annotation name the plugin is looking for versus what's in my service account.

In the ECR credential provider's code (main.go), it's looking for an annotation named "eks.amazonaws.com/ecr-role-arn":

arn, ok := request.ServiceAccountAnnotations["eks.amazonaws.com/ecr-role-arn"]

However, my service account has a different annotation name:

  annotations:
    eks.amazonaws.com/role-arn:
  arn:aws:iam::820537372947:role/AllowPullSpecificRepoRole

I'm using eks.amazonaws.com/role-arn (IRSA) but the code is looking for eks.amazonaws.com/ecr-role-arn.

The plugin also tries to fall back to an environment variable if the annotation isn't found:

  if !ok {
      arn = os.Getenv("AWS_ECR_ROLE_ARN")
  }

IRSA is setting AWS_ROLE_ARN, not AWS_ECR_ROLE_ARN. This probably explains the "no arn provided" error. The plugin can't find the ARN because:

1. It's looking for a different annotation name
2. It's looking for a different environment variable

To fix this issue, I think we have two options:

1. Change the service account annotation to match what the plugin expects:
   annotations:
   eks.amazonaws.com/ecr-role-arn: arn:aws:iam::820537372947:role/AllowPullSpecificRepoRole

2. Or add the environment variable the plugin expects:
   env:

    - name: AWS_ECR_ROLE_ARN
      value: arn:aws:iam::820537372947:role/AllowPullSpecificRepoRole