Sign in Subscribe
hidden

How to Set Up IAM Roles for EKS Nodes and Manage Docker Config Secrets

Divyansh

• 7 min read

Table of Contents

Introduction

Running containerized applications on Amazon EKS requires pulling images from private registries like ECR, Docker Hub, or other repositories. This needs proper authentication at both the node level and Kubernetes level.

This guide covers two essential EKS security components:

  1. IAM Role Configuration: Setting up IAM roles for EKS worker nodes to access AWS services
  2. Docker Config Secrets: Managing authentication credentials for private container registries

Without proper setup, pods fail to pull images and nodes lack permissions to interact with AWS services like ECR, CloudWatch, or other resources.

Understanding the Problem

EKS clusters face two main authentication challenges:

  1. Node-level authentication: Worker nodes need IAM permissions to access AWS services
  2. Container-level authentication: Pods need credentials to pull images from private registries

Let's visualize this architecture:

Introduction

Running containerized applications on Amazon EKS requires pulling images from private registries like ECR, Docker Hub, or other repositories. This needs proper authentication at both the node level and Kubernetes level.

This guide covers two essential EKS security components:

  1. IAM Role Configuration: Setting up IAM roles for EKS worker nodes to access AWS services
  2. Docker Config Secrets: Managing authentication credentials for private container registries

Without proper setup, pods fail to pull images and nodes lack permissions to interact with AWS services like ECR, CloudWatch, or other resources.

Understanding the Problem

EKS clusters face two main authentication challenges:

  1. Node-level authentication: Worker nodes need IAM permissions to access AWS services
  2. Container-level authentication: Pods need credentials to pull images from private registries

Let's visualize this architecture:

IAM Role Configuration for EKS Nodes

Understanding EKS Node IAM Requirements

EKS worker nodes need specific IAM permissions to function properly. The node role must allow the EC2 service to assume it and provide access to essential AWS services. Unlike traditional EC2 instances, EKS nodes require additional permissions for container networking, image pulling, and cluster communication.

The core requirements include:

  • EKS Worker Node Policy: Basic cluster communication and node registration
  • EKS CNI Policy: Container networking and IP management
  • ECR Read Only Policy: Pulling container images from Amazon ECR
  • CloudWatch Agent Policy: Logging and monitoring capabilities

Creating the Node Role

# Create trust policy and role
aws iam create-role --role-name EKSNodeRole \
    --assume-role-policy-document '{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"Service": "ec2.amazonaws.com"},
            "Action": "sts:AssumeRole"
        }]
    }'

# Attach essential policies
aws iam attach-role-policy --role-name EKSNodeRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name EKSNodeRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
aws iam attach-role-policy --role-name EKSNodeRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

# Create instance profile
aws iam create-instance-profile --instance-profile-name EKSNodeInstanceProfile
aws iam add-role-to-instance-profile \
    --instance-profile-name EKSNodeInstanceProfile \
    --role-name EKSNodeRole

Custom IAM Policies for Production

For production environments, create custom policies with minimal permissions instead of using broad AWS managed policies. This follows the principle of least privilege and reduces security risks.

Key considerations for custom policies:

  • ECR Access: Limit to specific repositories and regions
  • CloudWatch: Restrict to necessary log groups and metrics
  • S3 Access: Only if your applications require object storage
  • Secrets Manager: For applications that need credential retrieval

Creating Docker Config Secrets

Understanding Docker Config Secrets

Docker config secrets in Kubernetes store authentication credentials for private container registries. These secrets are used by the kubelet to authenticate when pulling container images. The secret contains a base64-encoded JSON structure that mirrors Docker's config.json format.

The secret type kubernetes.io/dockerconfigjson is specifically designed for this purpose and is automatically recognized by Kubernetes when pulling images.

Registry Authentication Methods

Different registries require different authentication approaches:

Docker Hub: Uses username/password or personal access tokens Amazon ECR: Uses temporary tokens obtained via AWS CLI Google Container Registry: Uses service account keys or workload identity Azure Container Registry: Uses service principal or managed identity

Creating Secrets for Different Registries

# Docker Hub with personal access token
kubectl create secret docker-registry dockerhub-secret \
    --docker-server=https://index.docker.io/v1/ \
    --docker-username=your-username \
    --docker-password=your-token

# Amazon ECR (token expires every 12 hours)
kubectl create secret docker-registry ecr-secret \
    --docker-server=123456789012.dkr.ecr.us-west-2.amazonaws.com \
    --docker-username=AWS \
    --docker-password=$(aws ecr get-login-password --region us-west-2)

# Custom registry with YAML
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: custom-registry-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: $(echo -n '{"auths":{"registry.example.com":{"auth":"'$(echo -n 'user:pass' | base64)'"}}}' | base64 -w 0)
EOF

Secret Management Best Practices

Credential Rotation: ECR tokens expire every 12 hours, requiring regular updates. Consider using external secret operators like External Secrets Operator or AWS Secrets Manager CSI driver for automatic rotation.

Namespace Isolation: Create secrets in specific namespaces rather than default to limit access scope.

RBAC Integration: Use Kubernetes RBAC to control who can access and modify these secrets.

Monitoring: Set up alerts for failed image pulls that might indicate credential issues.

Node Group Configuration

EKS Node Group Architecture

EKS node groups are managed groups of EC2 instances that run your containerized applications. Each node group must be associated with an IAM role that provides the necessary permissions for the nodes to join the cluster and access AWS services.

Key considerations for node group configuration:

  • Instance Types: Balance between cost and performance based on workload requirements
  • Scaling Configuration: Set appropriate min/max/desired sizes for your traffic patterns
  • Subnet Placement: Use private subnets for security, public subnets for cost optimization
  • AMI Type: AL2_x86_64 for standard workloads, AL2_ARM_64 for ARM-based applications

Creating Node Groups

# Basic node group creation
aws eks create-nodegroup \
    --cluster-name my-cluster \
    --nodegroup-name main-nodes \
    --node-role arn:aws:iam::123456789012:role/EKSNodeRole \
    --instance-types t3.medium \
    --scaling-config minSize=1,maxSize=5,desiredSize=2 \
    --subnets subnet-12345 subnet-67890

Infrastructure as Code with Terraform

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "main"
  node_role_arn   = aws_iam_role.node_group.arn
  subnet_ids      = aws_subnet.private[*].id

  scaling_config {
    desired_size = 2
    max_size     = 5
    min_size     = 1
  }

  instance_types = ["t3.medium"]
  
  depends_on = [
    aws_iam_role_policy_attachment.node_group_AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.node_group_AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.node_group_AmazonEC2ContainerRegistryReadOnly,
  ]
}

Node Group Best Practices

Multi-AZ Deployment: Always deploy node groups across multiple availability zones for high availability.

Instance Diversity: Consider using multiple instance types in a single node group to improve availability and cost optimization.

Taint and Labels: Use node taints and labels to control pod scheduling and create dedicated node pools for specific workloads.

Auto Scaling: Configure cluster autoscaler to automatically adjust node group size based on resource demands.

Testing and Validation

Comprehensive Testing Strategy

Testing your EKS authentication setup requires validating both IAM permissions and docker config secrets. This ensures your cluster can properly authenticate with AWS services and pull images from private registries.

IAM Permission Validation

Test node-level IAM permissions by creating a pod that attempts to access AWS services:

apiVersion: v1
kind: Pod
metadata:
  name: iam-test
spec:
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command: ["sleep", "3600"]
  restartPolicy: Never

Docker Config Secret Testing

Validate registry authentication with a test pod:

apiVersion: v1
kind: Pod
metadata:
  name: registry-test
spec:
  containers:
  - name: test-container
    image: your-private-registry/your-image:latest
    command: ["sleep", "3600"]
  imagePullSecrets:
  - name: dockerhub-secret
  restartPolicy: Never

Validation Commands

# Verify secrets exist and are properly configured
kubectl get secrets
kubectl describe secret dockerhub-secret

# Check node IAM role assignment
kubectl get nodes -o wide

# Test AWS service access from within pods
kubectl exec -it iam-test -- aws sts get-caller-identity
kubectl exec -it iam-test -- aws ecr describe-repositories

Troubleshooting Common Issues

ImagePullBackOff Errors: Usually indicate authentication problems with private registries. Check secret configuration and credential validity.

Access Denied Errors: Often related to insufficient IAM permissions. Verify the node role has the required policies attached.

Token Expiration: ECR tokens expire every 12 hours. Implement automated token refresh for production environments.

Security Best Practices

IAM Security Principles

Least Privilege Access: Create custom IAM policies with minimal required permissions instead of using broad AWS managed policies. This reduces the attack surface and limits potential damage from compromised credentials.

Resource-Specific Permissions: Limit ECR access to specific repositories and regions rather than granting global access. This prevents unauthorized access to other team's container images.

Regular Access Reviews: Implement periodic reviews of IAM permissions to ensure they remain aligned with current requirements and remove unused access.

Secret Management Strategy

External Secret Management: Use AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault for production environments. These services provide automatic rotation, audit logging, and centralized management.

Credential Rotation: Implement automated credential rotation for ECR tokens and other time-sensitive credentials. Consider using External Secrets Operator for Kubernetes-native secret management.

Environment Isolation: Use separate secrets for different environments (dev, staging, production) to prevent cross-environment access.

Network and Pod Security

Network Policies: Implement Kubernetes network policies to restrict pod-to-pod communication and limit egress traffic to necessary destinations only.

Pod Security Standards: Enforce pod security standards to prevent privilege escalation and limit container capabilities.

Image Security: Use vulnerability scanning tools like Trivy or Clair to scan container images before deployment. Implement image signing and verification for critical applications.

Monitoring and Compliance

Audit Logging: Enable CloudTrail logging for all IAM and ECR operations to track access patterns and detect suspicious activity.

Security Monitoring: Set up alerts for failed authentication attempts, unusual access patterns, and credential expiration.

Compliance Frameworks: Align your security practices with industry standards like SOC 2, PCI DSS, or HIPAA depending on your compliance requirements.

Advanced Configuration Patterns

Multi-Region ECR Access

For organizations operating across multiple AWS regions, configure ECR access that spans regions while maintaining security boundaries. This requires careful IAM policy design to allow cross-region access without compromising security.

Key considerations:

  • Regional IAM Policies: Create region-specific policies that limit access to appropriate ECR repositories
  • Cross-Region Replication: Implement ECR cross-region replication for disaster recovery and performance optimization
  • Token Management: Handle ECR tokens across multiple regions with proper expiration handling

Workload Identity Integration

For applications running on EKS that need to access AWS services, consider implementing AWS IAM Roles for Service Accounts (IRSA). This provides more secure and granular access control compared to node-level IAM roles.

Benefits of IRSA:

  • Pod-Level Permissions: Each pod can have specific IAM permissions based on its service account
  • Credential Isolation: Eliminates the need to share node-level credentials across all pods
  • Audit Trail: Better tracking of which specific workloads access which AWS resources

GitOps Integration

Integrate your EKS authentication setup with GitOps workflows for automated deployment and configuration management. This ensures consistent security policies across environments and provides version control for your infrastructure configurations.

Implementation approaches:

  • ArgoCD/Flux Integration: Use GitOps tools to manage IAM roles and secrets through version-controlled manifests
  • Policy as Code: Implement security policies using tools like OPA Gatekeeper or Kyverno
  • Automated Testing: Include security validation in your CI/CD pipelines

Conclusion

Setting up IAM roles for EKS nodes and managing docker config secrets is essential for secure container deployments. Key points:

  1. IAM roles provide nodes with AWS service permissions
  2. Docker config secrets enable pods to pull images from private registries
  3. Security best practices ensure least privilege access and proper credential management

This guide provides a foundation for running containerized applications on EKS with proper authentication and authorization.

For production, implement additional security measures like pod security policies, network policies, and regular credential rotation.

Whether you're looking to secure an existing EKS cluster, implement container registry authentication, or migrate to a more secure Kubernetes architecture, our team delivers proven solutions that enhance security while reducing operational complexity.
Visit kubenine.com to learn more about our DevOps consulting services and how we can help secure your EKS deployment.

Recent Posts