Kubernetes is a container orchestration tool that reduces the complexity of deploying and managing containerized applications. It’s fast becoming the industry standard for the deployment and management of large-scale applications and microservices, and is widely used by a diverse group of organizations to manage their applications, both in the cloud and on-premises.
As a Kubernetes administrator, it’s important you have a record of events that happen in your cluster. These records will serve as a source of truth for debugging issues and improving your cluster’s security. Kubernetes auditing documents actions that were executed—or actions that someone attempted to execute—in your cluster.
In this article, you’re going to learn what Kubernetes audit logs are, why they’re important, and how to enable audit logs in your Kubernetes cluster.
Understanding Kubernetes audit logs
Audit logs are a set of records that contain a chronological list of all requests made to the Kubernetes API. Kubernetes stores the actions generated by each user, as well as by the control plane. These logs are in JSON format, and contain information such as the HTTP method, information of the user who initiated the request, the requests that were initiated, the Kubernetes component handling the request, and much more.
The audit logs can tell you what happened and when, as well as who initiated it, where the request was observed, where it was initiated, and where it was going.
Why Kubernetes audit logs are important
Audit logs provide a comprehensive overview of everything happening in your cluster, allowing you to make informed decisions. It’s important to note that improperly configured audit logs can contain so much information that they’re impossible to use, so it’s important to make sure that your configuration gives you enough information to get the logs you need, but not so much that it’s overwhelming.
Audit logs can also provide visibility into what an attacker is trying to do—or has already done—in your cluster. When a user initiates a request to the API server, the API server checks that the user has sufficient permissions before it processes the request; if the user doesn’t have authorization, the API server rejects or forbids the request. In either case, Kubernetes saves both the request and the subsequent execution or rejection in the audit logs. In a scenario where an attacker hijacks a user or service account, you can view all the actions they performed or tried to perform.
What are the use cases for audit logs?
The following is an example of a Kubernetes audit log. Each key in the JSON structure contains information that’s vital to understanding what’s happening in your cluster.
Now you’ll take a closer look at the purpose of some of the keys and how they apply to your Kubernetes cluster.
User information and originating IP
The user key tells you what user or service account initiated the request through the Kubernetes API server, and the sourceIP key provides the IP of that account. An IP address can tell you the location (such as city, ZIP code, or area code), and the name of the user’s ISP. IP addresses aren’t always reliable, because they can be altered or hidden by the user making the request, but this information can be useful when trying to block malicious activities originating from a range of IP addresses.
Information about requests that were initiated and executed
The verb, requestURI, and objectRef all provide information about the requests users initiated in the cluster. The verb key indicates the Kubernetes HTTP request method that was performed: get, post, list, watch, patch, or delete. The requestURI provides information about the API requests you initiated in the cluster—for example, a request to get all pods, or create new deployments. The objectRef contains information (such as the name, namespace, apiVersion) about the Kubernetes object associated with the request. The objectRef, verb, and requestURI provide full information about the request initiated by the user.
Response status of the actions performed
The responseStatus and responseObject, in combination with the annotations key, provide insight into the response to the requests initiated by users or service accounts. The annotations.authorization.k8s.io/decision provides either an allow or reject value. These keys are very useful when performing audits on your Kubernetes cluster to detect anomalous behaviors.
Enabling and accessing audit logs
Now that you know what audit logs look like and why you need them, you’ll learn about how you can enable and access them on your Kubernetes cluster.
Self-hosted Kubernetes cluster
Self-hosted Kubernetes clusters don’t have audit logs enabled by default. The following steps explain how to enable and view audit logs in your Kubernetes cluster. In this section, you’ll use Minikube to set up a Kubernetes cluster on your local machine so that you can follow along. Minikube is a free tool that allows you to run a single-node Kubernetes cluster on your local machine. It’s a platform to practice and learn Kubernetes, or to use for daily development work. It’s important to note that you can’t use Minikube to host production-based applications.
- Follow this guide to install
minikubeon your local machine. - Once installed, run the command below to start your cluster
Minikube creates a cluster consisting of the Kubernetes primary and worker components on your local machine.
To enable audit logs, you’ll have to perform some configuration, such as updating the kube-apiserver config file and creating the directory path for the log file. The kube-apiserver is part of the control plane components that are hosted on the primary node, which means that its config file is located in the primary node. Because of this, you have to SSH into the primary node to update its config file, as you can’t do it imperatively using the kubectl command.
- To access the
kube-apiserverconfig file in your minikube Kubernetes cluster, run the command below.
- Create an
audit-policy.yamlfile.
- Open the file using the Vim editor.
- Paste the following example rules into the file. This is a basic policy file modeled on the example in the Kubernetes documentation, and can be modified as indicated in the comments to suit your logging needs. Kubernetes uses this policy file to identify if events should be logged or excluded.
- Create
audit.login the following directory. This is where Kubernetes will save your audit logs.
- Run the command below to edit the
kube-apiserverconfig file.
- Update the volume mount section of the config file.
- Update the volumes section.
- Finally, update the command section to enable auditing in your cluster.
Test the auditing feature
Give Kubernetes a few seconds to apply the changes you made to the kube-apiserver config file, and you can test your auditing to make sure it works.
- Open a new terminal and run the command below to check that your API server is up and running.
- Create a new pod to see if Kubernetes will log the request in the log file
- Return to the
minikube sshterminal and open the log file to see if it has logs in it by running the command below.
- If you see content in the log file that’s similar to the image below, auditing is now enabled and configured on your Kubernetes cluster
Managed Kubernetes cluster by cloud providers
Cloud providers offering managed Kubernetes as a service provide advanced auditing for your Kubernetes cluster—that is, they’ve integrated Kubernetes auditing with their auditing and logging tools. Google Kubernetes Engine (GKE) enables audit logging out of the box, while Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) require that you enable logging before you can start viewing logs generated on your cluster.
To enable or disable audit logging on a managed Kubernetes service, follow the guides in the official documentation, linked below.
Final thoughts
Auditing is very important to the security of your Kubernetes cluster and, in this article, you’ve learned more about Kubernetes audit logs, the importance of enabling audit logs in your Kubernetes cluster, and how to enable and configure audit logs on both self-hosted and managed Kubernetes clusters.
You also learned how to supplement Kubernetes audit logs with third-party tools. If you're looking for a powerful third-party tool that makes it easy to monitor your Kubernetes cluster and provides strong built-in activity and audit log functions, then consider Airplane. Airplane is the developer platform for building internal tools. The basic building blocks of Airplane are Tasks, which are single or multi-step functions that anyone on your team can use. Airplane also offers Views, a React-based platform for building custom UIs.
To build your first monitoring dashboard in minutes and utilize Airplane's powerful audit log feature, sign up for a free account or book a demo.
