Amazon Elastic Kubernetes Service (Amazon EKS) Security best practices overview

EKS:

Kubernetes control plane consists of at least two API server instances and three etcd instances that run across three Availability Zones within a Region.

Amazon EKS:

• Actively monitors the load on control plane instances and automatically scales them to ensure high performance.
• Automatically detects and replaces unhealthy control plane instances, restarting them across the Availability Zones within the Region as needed.
• Leverages the architecture of AWS Regions in order to maintain high availability.

Security in Amazon EKS:

AWS is responsible for the Kubernetes control plane, which includes the control plane nodes and etcd database. Third-party auditors regularly test and verify the effectiveness of our security as part of the AWS compliance programs.

Your responsibility includes the following areas.

• The security configuration of the data plane, including the configuration of the security groups that allow traffic to pass from the Amazon EKS control plane into the customer VPC
• The configuration of the nodes and the containers themselves
• The node’s operating system (including updates and security patches)
• Other associated application software:
• Setting up and managing network controls, such as firewall rules
• Managing platform-level identity and access management, either with or in addition to IAM
• The sensitivity of your data, your company’s requirements, and applicable laws and regulations

Amazon EKS identity-based policy best practices:

• Get started using AWS managed policies
• Enable MFA for sensitive operations
• Use policy conditions for extra security
• Don’t use a service account token for authentication
• Employ least privileged access to AWS Resources
• Use IAM Roles when multiple users need identical access to the cluster
• Employ least privileged access when creating RoleBindings and ClusterRoleBindings
• Make the EKS Cluster Endpoint private
• Create the cluster with a dedicated IAM role
• Regularly audit access to the cluster
• Alternative Approaches to Authentication and Access Management

Pods have a variety of different settings that can strengthen or weaken your overall security posture.

• Restrict the containers that can run as privileged
• Do not run processes in containers as root
• Never run Docker in Docker or mount the socket in the container
• Restrict the use of hostPath or if hostPath is necessary restrict which prefixes can be used and configure the volume as read-only
• Set requests and limits for each container to avoid resource contention and DoS attacks
• Do not allow privileged escalation
• Disable ServiceAccount token mounts
• Disable service discovery
• Configure your images with read-only root file system

Namespaces and Role-based access controls (RBAC) can be implemented to logically isolate tenants from each other. Following constructs are used to isolate tenants from each other:

• Namespaces
• Namespaces are fundamental to implementing soft multi-tenancy. They allow you to divide the cluster into logical partitions. Quotas, network policies, service accounts, and other objects needed to implement multi-tenancy are scoped to a namespace.
• Network policies
• By default, all pods in a Kubernetes cluster are allowed to communicate with each other. This behavior can be altered using network policies.
• Network policies restrict communication between pods using labels or IP address ranges. In a multi-tenant environment where strict network isolation between tenants is required, we recommend starting with a default rule that denies communication between pods, and another rule that allows all pods to query the DNS server for name resolution. With that in place, you can begin adding more permissive rules that allow for communication within a namespace. This can be further refined as required.
• Role-based access control (RBAC)
• Roles and role bindings are the Kubernetes objects used to enforce role-based access control (RBAC) in Kubernetes. Roles contain lists of actions that can be performed against objects in your cluster. Role bindings specify the individuals or groups to whom the roles apply. In the enterprise and KaaS settings, RBAC can be used to permit administration of objects by selected groups or individuals.
• Quotas
• Quotas are used to define limits on workloads hosted in your cluster. With quotas, you can specify the maximum amount of CPU and memory that a pod can consume, or you can limit the number of resources that can be allocated in a cluster or namespace. Limit ranges allow you to declare minimum, maximum, and default values for each limit.
• Pod priority and pre-emption
• Pod priority and pre-emption can be useful when you want to provide different qualities of services (QoS) for different customers.

Network policies:

Kubernetes network policies give you a mechanism to restrict network traffic between Pods (often referred to as East/West traffic) and between Pods and external services. Kubernetes network policies operate at layers 3 and 4 of the OSI model. Network policies use pod selectors and labels to identify source and destination pods, but can also include IP addresses, port numbers, protocol number, or a combination of these.

• Create a default deny policy
• Create a rule to allow DNS queries
• Incrementally add rules to selectively allow the flow of traffic between namespaces/pods
• Log network traffic metadata
• Use encryption with AWS load balancers

Security groups:

EKS uses AWS VPC Security Groups (SGs) to control the traffic between the Kubernetes control plane and the cluster’s worker nodes. Security groups are also used to control the traffic between worker nodes, and other VPC resources, and external IP addresses.

Amazon EKS clusters with Kubernetes version 1.13 and higher have a default pod security policy named eks.privileged.
This policy has no restriction on what kind of pod can be accepted into the system, which is equivalent to running Kubernetes with the PodSecurityPolicy controller disabled.
This security group allows unfettered communication between the EKS control plane and the nodes from managed node groups. For simplicity, it is recommended that you add the cluster SG to all node groups, including unmanaged node groups.

Encryption in transit:

The following are a few ways that you can encrypt data in a Kubernetes environment:

• Nitro Instances
• Traffic exchanged between the following Nitro instance types C5n, G4, I3en, M5dn, M5n, P3dn, R5dn, and R5n, is automatically encrypted by default.
• Container Network Interfaces (CNIs)
• Service Mesh
• Encryption in transit can also be implemented with a service mesh like App Mesh, Linkerd v2, and Istio.
• App Mesh also supports TLS encryption with a private certificate issued by AWS Certificate Manager (ACM) or a certificate stored on the local file system of the virtual node.
• Ingress Controllers and Load Balancers
• Ingress controllers are a way for you to intelligently route HTTP/S traffic that emanates from outside the cluster to services running inside the cluster. Oftentimes, these Ingresses are fronted by a layer 4 load balancer, like the Classic Load Balancer or the Network Load Balancer (NLB).

Encryption at rest:

There are three different AWS-native storage options you can use with Kubernetes: EBS, EFS, and FSx for Lustre. All three offer encryption at rest using a service managed key or a customer master key (CMK).

• Encrypt data at rest
• Rotate your CMKs periodically
• Use EFS access points to simplify access to shared datasets

Secrets management:

To show secrets from Secrets Manager and parameters from Parameter Store as files mounted in Amazon EKS pods, you can use the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver. The ASCP works with Amazon Elastic Kubernetes Service (Amazon EKS) version 1.17 or later.

Kubernetes secrets are used to store sensitive information, such as user certificates, passwords, or API keys. They are persisted in etcd as base64 encoded strings. On EKS, the EBS volumes for etcd nodes are encrypted with EBS encryption.

• Use AWS KMS for envelope encryption of Kubernetes secrets
• Audit the use of Kubernetes Secrets
• Rotate your secrets periodically
• Use separate namespaces as a way to isolate secrets from different applications
• Use volume mounts instead of environment variables
• Use an external secrets provider

Image security

You should consider the container image as your first line of defense against an attack.

• Create minimal images
• Use multi-stage builds
• Scan images for vulnerabilities regularly
• Create IAM policies for ECR repositories
• Consider using ECR private endpoints
• Implement endpoint policies for ECR
• Create a set of curated images
• Add the USER directive to your Dockerfiles to run as a non-root user
• Lint your Dockerfiles
• Build images from Scratch
• Use immutable tags with ECR
• Sign your images
• Update the packages in your container images

Security is a critical consideration for configuring and maintaining Kubernetes clusters and applications. The Center for Internet Security (CIS) Kubernetes Benchmark provides guidance for Amazon EKS node security configurations. The benchmark:

• Is applicable to Amazon EC2 nodes (both managed and self-managed) where you are responsible for security configurations of Kubernetes components.
• Provides a standard, community-approved way to ensure that you have configured your Kubernetes cluster and nodes securely when using Amazon EKS.
• Consists of four sections; control plane logging configuration, node security configurations, policies, and managed services.
• Supports all of the Kubernetes versions currently available in Amazon EKS and can be run using kube-bench, a standard open source tool for checking configuration using the CIS benchmark on Kubernetes clusters.

Logging and monitoring in Amazon EKS:

Amazon EKS control plane logging provides audit and diagnostic logs directly from the Amazon EKS control plane to CloudWatch Logs in your account.
You can select the exact log types you need, and logs are sent as log streams to a group for each Amazon EKS cluster in CloudWatch.

The following cluster control plane log types are available:

• Kubernetes API server component logs (api)
• Audit (audit)
• Authenticator (authenticator)
• Controller manager (controllerManager)
• Scheduler (scheduler)

Amazon EKS is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Amazon EKS.
CloudTrail is enabled on your AWS account when you create the account. When activity occurs in Amazon EKS, that activity is recorded in a CloudTrail event along with other AWS service events in Event history.

Perform the following actions as a line of secure logging:

• Enable audit logs
• Instructions for enabling/disabling the control plane logs, which includes the logs for the Kubernetes API server, the controller manager, and the scheduler, along with the audit log, can be found here.
• Utilize audit metadata
• Create alarms for suspicious events
• Analyze logs with Log Insights
• Audit your CloudTrail logs
• Use CloudTrail Insights to unearth suspicious activity

Compliance validation for Amazon EKS:

Third-party auditors assess the security and compliance of AWS services as part of multiple AWS compliance programs, such as SOC, PCI, FedRAMP, and HIPAA.

You can download third-party audit reports using AWS Artifact.

Your compliance responsibility when using AWS services is determined by the sensitivity of your data, your company’s compliance objectives, and applicable laws and regulations.

AWS provides the following resources to help with compliance:

• Security and Compliance Quick Start Guides — These deployment guides discuss architectural considerations and provide steps for deploying baseline environments on AWS that are security and compliance focused.
• Architecting for HIPAA Security and Compliance Whitepaper — This whitepaper describes how companies can use AWS to create HIPAA-compliant applications.
• AWS Compliance Resources — This collection of workbooks and guides might apply to your industry and location.
• Evaluating Resources with Rules in the AWS Config Developer Guide — The AWS Config service assesses how well your resource configurations comply with internal practices, industry guidelines, and regulations.
• AWS Security Hub — This AWS service provides a comprehensive view of your security state within AWS that helps you check your compliance with security industry standards and best practices.
• AWS Audit Manager — This AWS service helps you continuously audit your AWS usage to simplify how you manage risk and compliance with regulations and industry standards.

The line that delineates what AWS and its users are responsible for will vary depending on the service. For example, with Fargate, AWS is responsible for managing the physical security of its data centers, the hardware, the virtual infrastructure (Amazon EC2), and the container runtime (Docker). Users of Fargate are responsible for securing the container image and their application.

Knowing who is responsible for what is an important consideration when running workloads that must adhere to compliance standards.

The following table shows the compliance programs with which the different container services conform:

Lets Connect via LinkedIn — https://linkedin.com/in/hari-kiran-vusirikala/

Hari Kiran Vusirikala - Full Stack Developer - Infosys | LinkedIn