K8S Focus Areas

Layers and Phases¶

Layers¶


Applications	Manage sensitive data required by the application, as well as securing traffic flows and data within the application Kubernetes itself provides several abstractions to help manage application security
Clusters	A Kubernetes cluster consists of several control plane components, and components that run on worker nodes Understand how to secure Kubernetes and correctly configure Kubernetes components for each cluster
Infrastructure	Kubernetes components require compute, networking, and storage For Kubernetes, this corresponds to the nodes (virtual or physical hosts) that Kubernetes is installed on This layer must also be secured to ensure that Kubernetes components are correctly configured

Phases¶


Build	This phase involves the setup — before any workload is executed For applications, this phase includes build process and CI/CD pipeline concerns For clusters, and cluster add-on services, this includes the setup and configuration of Kubernetes For infrastructure, this includes the host prep process
Operate	This phase involves the ongoing operations and management of Kubernetes components, cluster add-on services, and workloads

Controls¶

	Build	Operate
Applications	Enforce Secure Images Dockerfile Linting Image Scanning	Image Provenance Secrets Management Namespaces Network Policies
Clusters	RBAC Audit Policies and Logging Certificate Management Pod Security Policies	Identity and Access Kubernetes Upgrades CIS benchmarks (k8s)
Infrastructure	Minimal OS OS hardening	CIS benchmarks (docker)

Applications¶

Control	Phase	Description
Enforce Secure Images	Build	See Docker Focus Areas
Dockerfile Linting	Build	Use specific tags Lint the Dockerfiles which ensures we are pulling base images from a specific tag, not just the generic “latest” tag Great Dockerfile linters are hadolint and dockerfilelint
Dockerfile Linting	Build	Whitelist base images Run a script that accepts Dockerfiles with FROM directives pointing to a smart whitelist of base images Rejects images that are pulling random untrusted images The whitelist is smart because it has a specific list of approved values but also has some resolution strategies/plans when the value is not in the list One of them is to check if the base image is a base Docker standard library image
Dockerfile Linting	Build	Linux Package Managers Use Linux distributions that check the integrity of the package using the package manager’s security features
Dockerfile Linting	Build	Sensitive Volumes Don’t allow images to be built whose Dockerfile specifies a sensitive host path as a volume mount (scan the Dockerfile for volume mounts like `/proc` or `/`) If a container like this was to be built and put into a Kubernetes cluster, there is an increased possibility that, if compromised, it could be used to expose information about the host and aid further penetration of the entire infrastructure
Dockerfile Linting	Build	Block root users Use linters and static code analysis to reject images that only specify the root user as the process which will execute the program inside of the container
Dockerfile Linting	Build	Squash Images Sometimes during image construction, you will need a private key or credentials to download all the associated resources required (e.g., private ruby gems) Unfortunately, when those keys or secrets are put into the container at build time, they are there hiding in the filesystem even when they aren’t needed for runtime When we absolutely need something like this we use the Docker directive COPY to bring it into the container, then after it has been used we use RUN rm … to remove it When you follow that procedure with — squash added to the docker build command the layer key which was just deleted will not end up in the final built image This means that the key or secret which was in the container previously is now gone permanently from all layers Once you push the squashed image, it is free of those files you would rather keep a secret
Dockerfile Linting	Build	Inspect Containers We are very explicit with the packages inside the container We run scripts that list everything inside the container on the build server. This forces the developers to see what’s inside their containers and can help them detect any packages they don’t need An example for displaying all packages installed on an rpm based distro would be: `docker exec $INSTANCE_ID rpm -qa`
Image Scanning	Build	Container images are typically built using orchestration tools, like Jenkins An image scanning tool needs to be part of the build process to scan each layer used in a container for vulnerabilities Clair is an open source image scanner, and CNCF backed image registries like Harbor use Clair to automatically scan all images Scan for malware and vulnerable packages, and reject images with high vulnerabilities One tool we use within the build that will prevent the pipeline from continuing → This allows us to prevent bad containers from deploying if a CVE is detected and published on it the same day One tool we use passively scans all images stored in the image registry → Will alert us if our images contain issues based on the CVE databases as they are updated over time (if the image didn’t have a CVE at the time of shipping, but had one later)
Image Provenance	Operate	While image scanning ensures that images that you build are safe, image provenance ensures that images that you run are the ones that you scanned and approved It is a way to ensure that only scanned and approved images are run in their clusters One way of doing that is provided a list of trusted image registries and using a cluster-wide policy management tool to ensure that images from non-trusted registries are not allowed Using Docker Content Trust, the build pipeline signs the metadata of a pushed image cryptographically so when it is pulled later, if the metadata on the image doesn’t match the decrypted metadata from the Notary server, the image is rejected at runtime
Secrets Management	Operate	Secrets are sensitive data, like password and keys, required by your application The best practice for managing secrets is to use “late-binding” and defer the loading of secrets from a secrets store to the application run-time — typically the initialization phase of the pod Example: Hashicorp Vault
Namespaces	Operate	Kubernetes Namespaces allow logical segmentation and isolation of resources, basically allowing one physical cluster to appear as several virtual clusters Whenever possible, applications should be isolated to their own namespaces This is important as several other Kubernetes features, such as RBAC, Resource Quotas, etc. can be applied at the namespace level However, it is important to note that namespaces do not automatically provide network isolation — this requires configuration of Network Policies
Network Policies	Operate	A Kubernetes Network Policy is like a firewall rule which allows fine-grained control of ingress and egress traffic to each application component i.e. a pod Kubernetes network policies should be configured at a Namespace level, for defaults, and at a workload level for each component Simply configuring Network Policies does nothing — a CNI that can enforce network policy rules, like Calico, is also needed

Clusters¶

Control	Phase	Description
Role-Based Access Controls (RBAC)	Build	Kubernetes provides granular role-based access controls (RBAC) capabilities to manage access to resources A Role defines a set of permission rules that specify which operations are allowed and on which entities A RoleBinding applies a role to a use identity or service account Both constructs, Role and RoleBinding, apply at a Namespace level. A ClusterRole and ClusterRoleBinding apply cluster-wide
Audit Policies and Logging	Build	A Kubernetes AuditPolicy defines what events need to be recorded and control what data should be included in the audit records The Audit Policy can be configured for different storage backends An Audit Policy and backends that record audit events must be configured for each Kubernetes cluster at the API Server level
Certificate Management	Build	Kubernetes components use X.509 certificates for authentication and encryption All Kubernetes certificates must be signed by a Certificate Authority (CA), however, the CA itself can be self-signed For an enterprise deployment, it is important to have a certificate management policy in place which ensures that Kubernetes certificates can be easily managed across clusters
Pod Security Policies	Build	Kubernetes Pod Security Policies manage rules for pod configuration and updates Pod Security Policies are cluster-wide resources and need to be enabled by the PodSecurityPolicy admission controller Simply creating a Pod Security Policy does nothing — each pods service account must be authorized to use it Pod Security Policies can control running or privileged containers, using the host network namespace, use of the host file-system and several other important privileges
Identity and Access	Operate	While Kubernetes RBAC provides granular control over access of entities, Kubernetes does not provide any construct to manage user identities This makes sense, as the best practices are to manage user identities via a central Identity Provider (IdP) such as Active Directory or other directory services In addition, it is important for enterprises to consider Single Sign-On (SSO) so that development and operations teams have a good user experience when managing multiple clusters across different infrastructure stacks or cloud providers
Kubernetes Upgrades	Operate	Kubernetes is a fast-moving project with minor feature releases every three months, and patches and security fixes released more often This means that enterprises need to be prepared to upgrade Kubernetes components often — on production clusters A managed Kubernetes service or a management tool that ensures safe and timely upgrades is required to operationalize Kubernetes
CIS Benchmarks for Kubernetes	Operate	The Center for Internet Security (CIS) publishes a list of over a hundred recommendations and best practices for securing Kubernetes clusters For secure operations, it’s essential to be able to audit clusters against the CIS benchmarks There are open source tools, like kube-bench from Aqua Security, that can help automate running the scans However, for production deployments, you will still need additional to collect, report, and analyze results

Infrastructure¶

Control	Phase	Description
Minimal OS	Build	Use stripped-down distributions for containers Reducing the operating system results in a smaller attack surface, and hence a more secure deployment
OS Hardening	Build	Most operating systems are insecure by default, and require hardening to minimize exposure to threats and vulnerabilities There are well-known procedures and standards for OS hardening, and these must be followed when building hosts that will run Kubernetes components
CIS Benchmarks for Docker	Operate	Kubernetes requires a container engine, like Docker CE or Containerd, to operate Container engines must also be secured and hardened As with securing Kubernetes clusters, the Center for Information Security (CIS) also publishes comprehensive benchmarks for securing container engines