The Linux operating system divides its memory into two different spaces: the kernel and the userspace. The kernel space contains the core of the operating system and has access to device drivers and system hardware. Due to its structure, the kernel space is only allowed to run secure and trusted code. On the other hand, the userspace contains regular applications and usually has restricted access to device drivers and system hardware. The userspace usually relies on the kernel space to perform operations like network I/O via device drivers and using system calls.
Sometimes, developers need the flexibility to implement new file systems and execute a secure custom system call without having to add code directly to the kernel space. That’s where the extended Berkeley Packet Filter (eBPF) comes in.
In this article, you’ll learn what eBPF is, why you need it, its various use cases, and how you can use it with Kubernetes for monitoring.
Why do you need eBPF?
If you have more than a passing familiarity with Kubernetes, you know that it relies heavily on Linux. eBPF is a Linux functionality that’s gaining traction in workflows with auditing, tracking, and monitoring.
eBPF is a kernel technology (starting from Linux 4.x) for Linux applications to securely execute code in Linux kernel space. It can be used to safely extend the capabilities of the kernel without changing the kernel source code.
Compared to strace, a tool for monitoring Linux processes and tracing system calls, eBPF provides system call tracing without too much overhead. Running the strace command on a server with tons of workloads can have catastrophic effects.
Another reason you might consider using eBPF is that a lot of big players are gradually adopting it for monitoring and observability of cloud-native applications. For example, Cilium is used for observing and monitoring connectivity between container workloads and is built with eBPF, and Cloudflare open-sourced a Prometheus exporter based on eBPF.
eBPF is widely used in a lot of open-source software in areas such as networking, monitoring, and creating security tools. Let’s take a closer look at some of its more common use cases when it comes to Kubernetes monitoring in particular:
- Advanced user-defined observability: With eBPF, you can run custom and user-defined programs in the kernel space to keep a record of network communications between services.
- Deep monitoring previously not possible in Kubernetes: eBPF can keep track of the latencies of a microservice as well the API endpoints in each service.
- Networking: The programmability of eBPF enables adding extra protocol parsers and easily programming any forwarding logic to meet changing requirements without ever leaving the packet processing context of the Linux kernel.
- Tracking TCP connections in Weave Scope: Weave Scope is a tool for monitoring, visualizing, and interacting with container-based systems.
- Pod and container-level network statistics: Control groups are usually known for grouping processes together hierarchically. In Kubernetes, this grouping happens at the container level. Since Linux 4.10, it’s now possible to attach eBPF programs to each control group. This helps with filtering network traffic from or to a process in the control group.
How does eBPF help with Kubernetes monitoring?
Services are usually deployed as self-contained entities called containers in a cluster. Since these containers are nothing but processes that run as long as the application in them is alive, eBPF provides a consistent tool for tracking the services running in a Kubernetes cluster.
You can also use eBPF for auditing commands executed in the Kubernetes cluster. eBPF can provide detailed visibility into each command, answering questions such as what happened, when did it happen, who initiated it, where did it happen, and so on. For example, it might be possible to see that a user executed kubectl exec to log in into a pod container, but difficult to see the exact parameters supplied like the pod name, namespace, and other relevant information. With eBPF, you can easily locate more information about such commands.
Since eBPF provides a unified framework for tracking processes, it offers more visibility into your system. It’s also quite secure. Unlike Linux kernel modules, eBPF allows you to execute secure code in the operating system kernel. Before these programs are loaded into the kernel space, they usually go through several checks to validate the code and ensure that it’s secure.
When does it make sense to use eBPF for monitoring?
There are a few scenarios where using eBPF for monitoring is a particularly solid choice.
- Monitoring from the kernel layer can mitigate the challenges relating to observability in the cloud. There are several issues that can result from using cloud observability, like request latencies. Some of these can be addressed by implementing eBPF.
- When there is a need for kernel tracing. eBPF allows you to execute custom code in the kernel. As everything is done in the kernel, tracing the actions and steps of the program becomes faster and more accurate. Also, the programs are executed when an event is triggered, so most actions are tracked.
- When traditional monitoring methods might not work. It’s possible for a monitoring tool to malfunction due to a bug. eBPF can offer visibility into requests and HTTP traffic even when traditional monitoring tools don’t work.
- When you need to optimize packet traffic control. In a standard Linux networking path, packets usually travel from source to destination. However, they’re not aware of complex paths. With eBPF and enough context, you can implement programs in the kernel that ensures the packet will avoid complex paths and follow the best and fastest route to the destination.
What are some limitations of eBPF?
The biggest limitation: eBPF is restricted to Linux and a recent kernel. It’s only available in newer versions of the Linux kernel.
eBPF has limited program functionality. The size of an eBPF program is limited to a maximum of 4096 bytes, which ensures the program terminates without an unbounded loop. This also limits the resources a program can have access to, and the functionality that can be implemented with the programs. Therefore, eBPF programs tend to be very small.
eBPF is not portable like other tracers, as it was solely developed for the Linux kernel and is completely dependent on it. eBPF programs are usually loaded at runtime by the kernel and the program is verified for possible threats. In the case of an unbounded loop, the verifier rejects the code, thus ensuring that all programs will terminate at some point. This limited support for loops may impair the capacity to create powerful networking programs.
Programs are also restricted to calling only an arbitrary set of kernel functions. They can only call the functions defined in the programs loaded into memory. That means the available helper functions that can be accessed by the programs at runtime are limited. While it’s possible to also read from kernel memory when tracing the path of a process using helpers like bpf_probe_read() and bpf_probe_read_str(), networking programs are not allowed to read arbitrary kernel memory since they don’t have access to that.
eBPF tools
There are several open-source tools you can use to build custom programs that get loaded into the kernel at runtime, in case you want to get your hands dirty. The list includes:
- BPF Compiler Collection (BCC) provides a toolkit for building efficient kernel manipulation programs.
- bpftrace is a high-level tracing language for eBPF programs.
- There are also language-specific tools like gobpf for Golang, libbpf for C/C++, and redbpf for Rust.
Final thoughts
eBPF is still an emerging technology, but companies are gradually adopting it in cloud-native applications running in production due to how it can improve monitoring and observation of those applications.
If you're looking to build a dashboard to help monitor and troubleshoot your applications, then try Airplane. Airplane is the developer platform for building internal tools. You can build custom UIs and workflows using SQL queries, Python scripts, Javascript functions, REST API calls, and more. You can utilize Airplane's pre-built component library and template library to build a monitoring dashboard efficiently.
To build your first internal dashboarding using Airplane, sign up for a free account or book a demo.
