Skip to content

K8S Focus Areas

Layers and Phases

Layers

Applications
  • Manage sensitive data required by the application, as well as securing traffic flows and data within the application
  • Kubernetes itself provides several abstractions to help manage application security
Clusters
  • A Kubernetes cluster consists of several control plane components, and components that run on worker nodes
  • Understand how to secure Kubernetes and correctly configure Kubernetes components for each cluster
Infrastructure
  • Kubernetes components require compute, networking, and storage
  • For Kubernetes, this corresponds to the nodes (virtual or physical hosts) that Kubernetes is installed on
  • This layer must also be secured to ensure that Kubernetes components are correctly configured

Phases

Build
  • This phase involves the setup — before any workload is executed
  • For applications, this phase includes build process and CI/CD pipeline concerns
  • For clusters, and cluster add-on services, this includes the setup and configuration of Kubernetes
  • For infrastructure, this includes the host prep process
Operate
  • This phase involves the ongoing operations and management of Kubernetes components, cluster add-on services, and workloads

Controls

Build Operate
Applications
  • Enforce Secure Images
  • Dockerfile Linting
  • Image Scanning
  • Image Provenance
  • Secrets Management
  • Namespaces
  • Network Policies
Clusters
  • RBAC
  • Audit Policies and Logging
  • Certificate Management
  • Pod Security Policies
  • Identity and Access
  • Kubernetes Upgrades
  • CIS benchmarks (k8s)
Infrastructure
  • Minimal OS
  • OS hardening
  • CIS benchmarks (docker)

Applications

Control Phase Description
Enforce Secure Images Build
Dockerfile Linting Build
    Use specific tags
    • Lint the Dockerfiles which ensures we are pulling base images from a specific tag, not just the generic “latest” tag
    • Great Dockerfile linters are hadolint and dockerfilelint
Dockerfile Linting Build
    Whitelist base images
    • Run a script that accepts Dockerfiles with FROM directives pointing to a smart whitelist of base images
    • Rejects images that are pulling random untrusted images
    • The whitelist is smart because it has a specific list of approved values but also has some resolution strategies/plans when the value is not in the list
      • One of them is to check if the base image is a base Docker standard library image
Dockerfile Linting Build
    Linux Package Managers
    • Use Linux distributions that check the integrity of the package using the package manager’s security features
Dockerfile Linting Build
    Sensitive Volumes
    • Don’t allow images to be built whose Dockerfile specifies a sensitive host path as a volume mount (scan the Dockerfile for volume mounts like /proc or /)
    • If a container like this was to be built and put into a Kubernetes cluster, there is an increased possibility that, if compromised, it could be used to expose information about the host and aid further penetration of the entire infrastructure
Dockerfile Linting Build
    Block root users
    • Use linters and static code analysis to reject images that only specify the root user as the process which will execute the program inside of the container
Dockerfile Linting Build
    Squash Images
    • Sometimes during image construction, you will need a private key or credentials to download all the associated resources required (e.g., private ruby gems)
    • Unfortunately, when those keys or secrets are put into the container at build time, they are there hiding in the filesystem even when they aren’t needed for runtime
    • When we absolutely need something like this we use the Docker directive COPY to bring it into the container, then after it has been used we use RUN rm … to remove it
    • When you follow that procedure with — squash added to the docker build command the layer key which was just deleted will not end up in the final built image
      • This means that the key or secret which was in the container previously is now gone permanently from all layers
      • Once you push the squashed image, it is free of those files you would rather keep a secret
Dockerfile Linting Build
    Inspect Containers
    • We are very explicit with the packages inside the container
    • We run scripts that list everything inside the container on the build server. This forces the developers to see what’s inside their containers and can help them detect any packages they don’t need
    • An example for displaying all packages installed on an rpm based distro would be: docker exec $INSTANCE_ID rpm -qa
Image Scanning Build
  • Container images are typically built using orchestration tools, like Jenkins
  • An image scanning tool needs to be part of the build process to scan each layer used in a container for vulnerabilities
  • Clair is an open source image scanner, and CNCF backed image registries like Harbor use Clair to automatically scan all images
  • Scan for malware and vulnerable packages, and reject images with high vulnerabilities
    • One tool we use within the build that will prevent the pipeline from continuing → This allows us to prevent bad containers from deploying if a CVE is detected and published on it the same day
    • One tool we use passively scans all images stored in the image registry → Will alert us if our images contain issues based on the CVE databases as they are updated over time (if the image didn’t have a CVE at the time of shipping, but had one later)
Image Provenance Operate
  • While image scanning ensures that images that you build are safe, image provenance ensures that images that you run are the ones that you scanned and approved
  • It is a way to ensure that only scanned and approved images are run in their clusters
  • One way of doing that is provided a list of trusted image registries and using a cluster-wide policy management tool to ensure that images from non-trusted registries are not allowed
  • Using Docker Content Trust, the build pipeline signs the metadata of a pushed image cryptographically so when it is pulled later, if the metadata on the image doesn’t match the decrypted metadata from the Notary server, the image is rejected at runtime
Secrets Management Operate
  • Secrets are sensitive data, like password and keys, required by your application
  • The best practice for managing secrets is to use “late-binding” and defer the loading of secrets from a secrets store to the application run-time — typically the initialization phase of the pod
  • Example: Hashicorp Vault
Namespaces Operate
  • Kubernetes Namespaces allow logical segmentation and isolation of resources, basically allowing one physical cluster to appear as several virtual clusters
  • Whenever possible, applications should be isolated to their own namespaces
    • This is important as several other Kubernetes features, such as RBAC, Resource Quotas, etc. can be applied at the namespace level
    • However, it is important to note that namespaces do not automatically provide network isolation — this requires configuration of Network Policies
Network Policies Operate
  • Kubernetes Network Policy is like a firewall rule which allows fine-grained control of ingress and egress traffic to each application component i.e. a pod
  • Kubernetes network policies should be configured at a Namespace level, for defaults, and at a workload level for each component
  • Simply configuring Network Policies does nothing — a CNI that can enforce network policy rules, like Calico, is also needed

Clusters

Control Phase Description
Role-Based Access Controls (RBAC) Build
  • Kubernetes provides granular role-based access controls (RBAC) capabilities to manage access to resources
  • Role defines a set of permission rules that specify which operations are allowed and on which entities
  • RoleBinding applies a role to a use identity or service account
  • Both constructs, Role and RoleBinding, apply at a Namespace level. A ClusterRole and ClusterRoleBinding apply cluster-wide
Audit Policies and Logging Build
  • A Kubernetes AuditPolicy defines what events need to be recorded and control what data should be included in the audit records
  • The Audit Policy can be configured for different storage backends
  • An Audit Policy and backends that record audit events must be configured for each Kubernetes cluster at the API Server level
Certificate Management Build
  • Kubernetes components use X.509 certificates for authentication and encryption
  • All Kubernetes certificates must be signed by a Certificate Authority (CA), however, the CA itself can be self-signed
  • For an enterprise deployment, it is important to have a certificate management policy in place which ensures that Kubernetes certificates can be easily managed across clusters
Pod Security Policies Build
  • Kubernetes Pod Security Policies manage rules for pod configuration and updates
  • Pod Security Policies are cluster-wide resources and need to be enabled by the PodSecurityPolicy admission controller
  • Simply creating a Pod Security Policy does nothing — each pods service account must be authorized to use it
  • Pod Security Policies can control running or privileged containers, using the host network namespace, use of the host file-system and several other important privileges
Identity and Access Operate
  • While Kubernetes RBAC provides granular control over access of entities, Kubernetes does not provide any construct to manage user identities
  • This makes sense, as the best practices are to manage user identities via a central Identity Provider (IdP) such as Active Directory or other directory services
  • In addition, it is important for enterprises to consider Single Sign-On (SSO) so that development and operations teams have a good user experience when managing multiple clusters across different infrastructure stacks or cloud providers
Kubernetes Upgrades Operate
  • Kubernetes is a fast-moving project with minor feature releases every three months, and patches and security fixes released more often
  • This means that enterprises need to be prepared to upgrade Kubernetes components often — on production clusters A managed Kubernetes service or a management tool that ensures safe and timely upgrades is required to operationalize Kubernetes
CIS Benchmarks for Kubernetes Operate
  • The Center for Internet Security (CIS) publishes a list of over a hundred recommendations and best practices for securing Kubernetes clusters
  • For secure operations, it’s essential to be able to audit clusters against the CIS benchmarks
  • There are open source tools, like kube-bench from Aqua Security, that can help automate running the scans
  • However, for production deployments, you will still need additional to collect, report, and analyze results

Infrastructure

Control Phase Description
Minimal OS Build
  • Use stripped-down distributions for containers
  • Reducing the operating system results in a smaller attack surface, and hence a more secure deployment
OS Hardening Build
  • Most operating systems are insecure by default, and require hardening to minimize exposure to threats and vulnerabilities
  • There are well-known procedures and standards for OS hardening, and these must be followed when building hosts that will run Kubernetes components
CIS Benchmarks for Docker Operate
  • Kubernetes requires a container engine, like Docker CE or Containerd, to operate
  • Container engines must also be secured and hardened
  • As with securing Kubernetes clusters, the Center for Information Security (CIS) also publishes comprehensive benchmarks for securing container engines