Kubernetes has changed DevOps since its release in 2014. Not only do you have access to hardware and resource allocation management as in a virtualized deployment, but now, you also have access to managed runtimes, updates, and portability. Kubernetes makes it easy to launch and scale without the headaches of orchestration.
Most organizations realize these benefits. In one survey, over 50% of participants reported that they used Kubernetes for container management, and the popularity of Kubernetes and managed Kubernetes services continues to grow at a rapid pace.
However, there’s a darker side to Kubernetes’ adoption. Over 94% of respondents in a RedHat survey said that they had suffered a container security incident. It’s important to consider this in context and to harden your Kubernetes deployment to preempt security issues like these.
In this article, I will share a brief overview of Kubernetes and its core components. Then, I’ll dive into the 4C framework for Kubernetes security and introduce some tactics for proactively addressing security vulnerabilities.
Kubernetes and its components
To understand security in Kubernetes, you need to understand how Kubernetes works at a high level. Each Kubernetes environment is loosely coupled and distributed so that if one environment is compromised, the others are relatively isolated from impact. however, there are still significant risks that need to be managed.
The two parts of a Kubernetes environment are the control plane and the node components. The control plane hosts the API server and Kube-controller managers, the key-value store, and the scheduler.
Within the control plane, there are four components connected through a hub and spoke pattern.
- Kube scheduler assigns Kube pods to nodes;
- Etc key-value store holds cluster data (including the config, state, and metadata);
- Kube-controller manager runs controller processes such as the jobs and node controller;
- Cloud-controller manager sets up connections with the cloud provider that you have selected.
The node components each include Kubelets and Kube-proxies:
- Kubelet ensures that containers run and are healthy;
- Kube-proxy maintains network rules on nodes, allowing communication to Pods from network sessions inside and outside the cluster.
With this architecture in mind, let’s explore some of the security best practices for Kubernetes environments.
Security architecture, concerns, and best practices
Although each Kubernetes component is separated, you still have to consider the communication between the control plane and the node worker and the security of the control plane itself. This is heavily influenced by your configurations and practices.
You can consider security in the context of the 4Cs of Kubernetes security: Cloud, Clusters, Containers, and Code.
You can think of these like a Matryoshka (or Russian Nesting Doll). Each layer (“C”) builds on and is vulnerable to the security issues of the layer that it is encompassed by. So, the code layer is vulnerable to security issues in the container, the clusters, and the cloud.
As a result, to secure the code and application data, all four layers - cloud, clusters, container, and code need to be secured.
Best practices for Kubernetes security
There are two main areas to secure in these four layers: the configurable cluster components and the applications which run in the cluster. In the following sections, I’ll cover some best practices in both of these areas for all four layers.
Code
I won’t cover all of the best practices for writing secure software code as this is a huge topic. But Kubernetes administrators primarily need to worry about network communication and attack prevention measures.
Network
The network describes how your code connects to outside services. You can reduce the vectors in which attackers can gain access to your systems by securing communication into and between your services.
Here are some recommendations:
- Network communications on TCP should be set up to perform a TLS handshake and have encryption by default. Third-party service meshes, which are run on each node, could be used for this; consider Istio, Linkerd, Consul), and Citrix.
- You should also encrypt communications between services.
- You should only expose ports that are essential for communication and network gathering.
Attack prevention
Alongside securing your communications, routine operation practices can be implemented which scan your code or internal configurations for vulnerabilities. A few tactics you might consider include:
- Schedule pen testing to preemptively defend against SQL injection, CSRF, and XSS attacks – a popular tool to address these in the open-source community is Project ZAP.
- Analyze code for vulnerabilities such as updates and changes to 3rd party dependencies and unsafe coding practices through at least one static analysis tool, such as kube-score.
Changes to libraries and bad code can introduce bugs and backdoors that could impact the security of your codebase. Automated monitoring can help with ensuring that the libraries are secure and common errors are detected prior to deployment.
Container
Your code, bundled with config files, libraries, and dependencies, is located in containers. There are a few important Kubernetes security considerations for containers:
- When choosing the container image/runtime, select runtime classes that have stronger isolation. Kubernetes does not protect against attacks in the runtime and doesn’t detect issues by default in the image.
- As part of the build step, scan for known vulnerabilities, and sign container images.
- Allow only essential communication between containers (in pods/clusters) and limit privileged access to specific requirements.
Using known containers from a private registry is also a good idea, but remember that images are often built on external base layers that might have vulnerabilities too.
Clusters
Clusters are a collection of containers within groups of pods. For clusters, it’s important to maintain security in the control plane, specifically:
- Restrict cluster access to the etcd backend (and only access etcd manually if you need to escalate permissions). A lot of sensitive data and permissions reside on the etcd, so you don’t want the rest of your cluster to be able to access it unless absolutely necessary.
- Enable audit logging and archive the audits on a secure server for monitoring and analysis during debugging.
- Encrypt secrets at rest so that adversaries can not access critical information if they penetrate your clusters’ defenses.
Cloud
Kubernetes clusters are typically run on cloud providers like Google, Amazon, and Microsoft. Each provider has documentation for their best practices which you should implement:
While your provider will take care of most high-level security issues in the cloud, you still have to pay attention to permissions and access controls. A few considerations that apply to all cloud hosting services include:
- Provide least privileged access to cloud resources.
- Make the auth role-specific and audit access to the resource through sources such as kubectl-who-can: https://github.com/aquasecurity/kubectl-who-can
- Never nest Docker in Docker or run processes as root.
- Regularly run kube-bench to comply with CIS Kubernetes benchmarks.
If you’re not using the cloud but instead relying on in-house servers, security at this layer gets more complicated as you’ll also need to consider things like physical security and failover.
Secure Kubernetes base configuration
When getting started with Kubernetes, base configurations are suggested for your environment. However, these are not intrinsically secure. There are several things you can do to further secure your Kubernetes base configuration, including:
- Monitor the source of your images and be diligent about third-party image registries. DockerHub is a good place to find official images.
- Define network policies carefully. They allow you to specify which entities internal and external services can connect with, but ingress and egress rules are not defined by default.
- Keep user security privileges on an as-needed basis with tight control over access and well-defined roles.
- Label both critical issues and identified security issues with unknown fixes in your security scanner of choice; once a fix is released, refer back to your identified security issues.
The role of monitoring and logging in security
It’s tough to implement all of these recommendations quickly. Keeping a critical eye on the defaults in Kubernetes and the 4C levels requires an ever-growing list of considerations. Even if you follow all of these best practices, new vulnerabilities are introduced by internal and external changes constantly, so the challenge never really ends.
This leads to my final recommendation, which is to implement robust logging and monitoring. With access to your Kubernetes events, logs, and alerts, you can mitigate issues before they spiral out of control.
As you continue to integrate Kubernetes into your growing organization, do your best to not be part of the 94% of organizations that experienced security breaches in their deployments. Consider the security best practices above, and keep an eye out for changes as Kubernetes is an actively evolving ecosystem.
If you're looking for a powerful platform that makes it easy to monitor your applications and troubleshoot them, consider using Airplane. Airplane is the developer platform for building internal tools. With Airplane, you can build Tasks, which are single or multi-step functions that anyone can use, and Views, which are React-based custom UIs, using scripts, queries, APIs, and more. Airplane also offers powerful built-ins, like activity and audit logs, that make your internal tools secure and efficient.
To try it out, sign up for a free account or book a demo.
