AWS Authentication for K8s: EKS Pod Identities Deep Dive

In the last article, I covered in brief, the solutions for the problem of authenticating into AWS from K8s workloads. This is the first detailed article (and the video) I promised I’ll follow up that with, going into to details on EKS Pod Identities. I’m hoping to make this the most detailed, most in-depth video on the topic, but this article would probably have less details, so I highly recommend watching the video. It also has a full demo on how to create an EKS cluster for Pod Identities.

Introduction

EKS Pod Identities are only usable in fully fledged EKS clusters, backed by EC2 compute. They are easy to setup, and easy to use, and is the recommended way to get short term credentials into K8s workloads.

You just,

Create the IAM Role to use
Create the K8s Service Account to map to the Role
Map the Serivce Account to IAM Role in AWS
Start using the Service Account for Pods

EKS takes care of providing proper credentials by doing everything in between the Pod and AWS.

How do Pod Identities Work?

Most details here are a bit hard to put together in a meaningful manner as a blog post. The video format works better here to explain details jumping between the demo and the architecture diagram.

If the details here are not that clear, make sure to check my video out. I go into more details than the blog post in the video.

The architecture of how Pod Identities work behind the scenes is a bit complex, compared to how easy it is to get it working as an end user. AWS does a pretty good job of hiding this complexity, but it is still interesting explore, how mostly standard approaches are taken to implement such an elegant UX (DX?) in this area.

I found most of these details by going through the documentation, the code for these components that are open source, and then some details from github issues that talk about a few undocumented scenarios.

There are several moving parts involved in Pod Identities.

EKS Pod Identity Addon

When you create an EKS cluster, you can add several addons to it to enable different types of functionality. EKS Pod Identity Addon is one of these, that in the background adds a Daemonset to the cluster to run an Agent, called the EKS Pod Identity Agent.

The addon also configures the EKS Pod Identity Webhook to configure pods in a specific way.

AWS IAM Roles

IAM Role to use for the specific workload needs to be created with the same best practices that you would use for any other scenario, Principle of Least Privilege, no Confused Deputies, etc.

Additionally, it should allow EKS to assume to role, or more specifically, the principal pods.eks.amazonaws.com to assume to it. This would be the principal EKS uses when a Pod is created to give it the temporary credentials.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

K8s Service Accounts and Mapping to IAM Role

Service Accounts are going to be used to map a Role to a given Pod. Essentially, a Service Account is mapped to the above IAM Role, which tells EKS to use the specific Role to generate temporary credentials.

❯ kubectl create sa pod-identity-test

❯ aws eks create-pod-identity-association --cluster-name <cluster-name>
--service-account pod-identity-test --namespace default --role-arn <role-arn>

AWS SDK Crdentials Lookup Chain

The AWS SDK, when trying to make an authenticated call to AWS, will try to locate credentials from a list of sources. This is called the credentials provider chain and contains source like the credentials file, environment variables, and Instance Metadata Service (IMDSv2). The provider of most interest in EKS Pod Identities is called the Container Credentials Provider, which is basically a way to get credentials from an external credentials server.

The SDK will source details for these different providers through different environment variables. For Container Credentials provider, the following two environment variables should be found.

spec:
  containers:
  - ...
    env:
    - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
      value: https://169.254.170.23/v1/credentials
    - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: amazon/aws-cli:latest
    imagePullPolicy: Always
    name: pod1
    ...

The values I’ve used above are the default values you would see in EKS Pod Identities scenario. There’s a process that runs on the link local address 169.254.170.23 on every Node in the cluster, and the token to provide to it (and all the way through to AWS through it) is in the Service Account token in /var/run/secrets/eks.amazonaws.com/serviceaccount/token.

So these environment variables should be set for every Pod that needs temporary credentials through EKS Pod Identities. This can be done manually, but you have better things to worry about. There should be a component that,

Mounts the Service Account token
Set these environment variables during Pod creation.

EKS Pod Identity Webhook

This is a K8s Mutating Webhook implementation that gets invoked when a new Pod is scheduled to be created. A mutating webhook mutates the Pod definition, as the name suggests, and in this case, the Pod Identity Webhook mutates the Pod definition so that a couple of environment variables and the Service Account token projection are added to it.

The Webhook knows the list Service Accounts to do these changes, by looking at a configuration file it watches constantly. When Pod Identities are enabled as an Addon, (I assume that) EKS keeps editing this file whenever a Service Account Role mapping is created, using the eks create-pod-identity-association command.

The environment variables injected by the Webhook, points the SDK to perform a credentials lookup from a credentials server, and proceeds to use those credentials to authenticate the requests to AWS. This credentials server is the EKS Pod Identity Agent.

EKS Pod Identity Agent

The Agent (which is not the Webhook, I know, they are confusing) is a credentials server that sits on every Node as a Daemonset. When it receives a request for credentials, along with a Service Account token, it will pass the token along to EKS to exchange it to temporary credentials.

EKS

When the EKS Pod Identity Agent requests a credentials exchange, EKS performs an AssumeRoleForPodIdentity operation to assume the target role and get temporary credentials. For this to work, the EKS Node Instance profile should have the permission eks-auth:AssumeRoleForPodIdentity attached to it. The easiest way to make sure this happens is to attach the AWS managed policy AmazonEKSWorkerNodePolicy to the Node role when creating it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "WorkerNodePermissions",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumesModifications",
                "ec2:DescribeVpcs",
                "eks:DescribeCluster",
                "eks-auth:AssumeRoleForPodIdentity"
            ],
            "Resource": "*"
        }
    ]
}

After this operation, the EKS Pod identity is returrned the temporary credentials which passes it back to the SDK on the Pod.

Conclusion

The complexity of the architecture is not really visible to the end user in EKS Pod Identities. They just have to,

Create the IAM Role to use
Create the K8s Service Account to map to the Role
Map the Serivce Account to IAM Role in AWS
Start using the Service Account for Pods

The approach recommended before EKS Pod Identities was introduced was equally complex in architecture and in how to set it up to get something out usefully. I will explore IAM Roles for Service Accounts (IRSA) in the next video.

Bibliography?

Links to documentation and code:

Credential Provider Chain - https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html#credentialProviderChain
Container Credentials Provider - https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html
Java SDK - https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core
AssumeRoleForPodIdentity docs - https://docs.aws.amazon.com/eks/latest/APIReference/API_auth_AssumeRoleForPodIdentity.html
TokenRequest docs - https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/
Pod Identity Webhook Github - https://github.com/aws/amazon-eks-pod-identity-webhook
Pod Identity Agent Github - https://github.com/aws/eks-pod-identity-agent