BPF applications let engineers get right under the hood of the operating system. These applications can monitor performance and access resources via hooks in kernel subsystems.
Working on kernel applications is a delicate balance, though. You need to ensure compatibility on a range of systems and avoid anything obsolete or deprecated. BCC can simplify that process, but libbpf has advantages that make it a better solution in many cases.
In this article you’ll see how libbpf differs from BCC, how to work with it, and what it offers in case you decide to make the switch. We’ll take a look at:
- BCC and its issues
- What is libbpf?
- Libbpf and bpf co-re
- Libbpf’s advantages over BCC
- How does libbpf work?
- Why would you use it?
- Getting started with libbpf
What is BPF?
Berkeley Packet Filter, or BPF, was originally a virtual machine that allowed programmers to access low-level kernel functions more safely and easily. It’s since evolved into a “generic kernel execution engine,” according to Netflix engineer Brendan Gregg. It ensures programs can’t crash or run indefinitely, which is a big change in how programmers interact with the operating system.
BPF’s efficiency means it’s being used to drive some of the biggest sites around. Facebook has over forty BPF programs running on every server, and sometimes as many as one hundred.
BCC is the established player for working with BPF, while libbpf is the new kid on the block.
BCC and its issues
BCC, or BPF Compiler Collection, uses extended BPFs to more easily create low-level programs. C is a common language choice, but Lua, Python, C++, and Rust are all viable alternatives.
It offers plenty of examples and tools along with useful error messages, helping to ensure your BPF programs are correct.
There are disadvantages to BCC, however. It embeds compiler components, such as Clang, and compiles programs at runtime. This takes extra CPU and memory and can expose issues that only appear when the program is compiled.
BCC needs kernel header packages, which means they must be installed on target hosts. That’s difficult if you’re using multiple machines. It also means type definitions for unexported kernel content need to be used in source code, which is messy.
What is Libbpf?
Libbpf is an alternate set of tools for building BPF applications. For networking, security, and analytic applications, it offers several potential advantages over BCC.
Libbpf and BPF CO-RE
Libbpf is usually used alongside BPF CO-RE (compile once, run everywhere). BPF CO-RE, which was designed to solve portability issues with BPF, allows you to create binaries that run on different kernel versions.
It includes BPF Type Format (BTF) information. This means you need to use a kernel build that has CONFIG_DEBUG_INFO_BTF=y set when compiled. If you’re using a standard consumer Linux build, you’ll need to do a custom compilation to enable this; otherwise, you’ll run into errors.
Libbpf’s advantages over BCC
Libbpf allows developers to focus on the task at hand by eliminating various headaches.
It produces simple binaries that compile once and run anywhere. It eliminates many dependencies and executes your code as close to one to one as possible.
The LLVM, kernel, and header dependencies that need to be installed to run programs compiled with BCC can run to over 100 MB. Eliminating the overhead from including LLVM and Clang libraries leads to smaller binaries.
For instance, one tool that included them compiled to 645 KB using BCC. A tool recompiled using libbpf tools resulted in a portable binary of just 151 KB. That’s a major size reduction.
Libbpf also creates binaries that use less memory, for example a 9 MB memory footprint compared to 80 MB for Python with BCC.
How does Libbpf work?
Libbpf acts as a BPF program loader. It loads, checks, and relocates BPF programs, sorting out maps and hooks. That frees developers to implement their programs without having to do all that housekeeping.
Offsets and types are matched and updated automatically, meaning a program can be run on a target host without the need for expensive additions such as Clang. You’re effectively writing a plain C user-mode program that does what you expect with no surprises.
One way libbpf achieves this is to use a vmheader file, which includes multiple kernel types so you’re not dependent on system-wide kernel headers. This means that to switch from BCC to libbpf, you need to include vmlinux.h.
A BPF application goes through several phases:
- Open Phase – The BPF program is paused while maps, variables, and global variables are discovered.
- Load Phase – Maps are created. BPF programs are loaded into the kernel and verified.
- Attachment Phase – BPF programs are attached to hooks, ready to work.
- Tear Down Phase – Resources are freed as BPF programs are detached and unloaded from the kernel.
If you look in minimal.c below you’ll see functions corresponding to each phase (destroy for the tear down phase).
The open and load phases can be combined if you don’t need to make runtime adjustments. Use this function:
You can also modify the attach phase to attach resources selectively if required.
Why should you use Libbpf?
Libbpf offers several benefits. Its lack of dependencies makes it quicker and easier to use on multiple machines. The more people you have working on your software, the greater this advantage will be.
It is better at resource usage, outputting smaller binaries, and using less memory, which makes it well-suited for system-critical tasks. Its limited impact on performance makes it ideal for monitoring, security, and analytics, too.
Getting started with Libbpf
To get started, try the libbpf-bootstrap demo applications on GitHub. After you download the repo, you can build the various examples using Make and Sudo. Here’s an example of building and testing some output for the Minimal demo:
This code and the one below come from the libbpf-bootstrap repo and use the BSD 3-Clause license.
Here’s an example of a libbpf hello world:
** minimal.bpf.c **:
And its companion file, minimal.c:
Further tips
Aside from Minimal, there are several other useful examples in the libbpf-bootstrap GitHub examples folder:
Bootstrap – This tracks processes and gives you stats on them, showing you how to do a few basic tasks. If you want to write a monitoring or performance tracking application, take a look here first.
Tracecon – Rust fans can take a look at tracecon, a Rust & Co. app that lets you track all the TCPv4 connections on the machine.
Uprobe – This shows you how to work with user-space probes, letting you track arguments and output.
Fentry – These are fentry- and fexit-based tracing programs. They perform better than kprobes, but require a kernel version of at least 5.5.
Kprobe – This feature is another logging example that works with kernel-space entry and exit probes.
XDP – This is a Rust example that logs packet sizes.
If you run into any issues with libbpf, take a look at its log output. It doesn’t use BCC’s method of removing memory limits to make sure programs can load into the kernel successfully.
To make sure you have enough memory, you can call setrlimit at the beginning of your program. You can see an example of that in minimal.c, above.
As these examples show, switching to libbpf isn’t too painful, and if you’re running into any of the above issues, the switch is well worth it.
Conclusion
BPF gives you observability superpowers, but there are issues with its portability and resource use. Libbpf helps you mitigate those problems, giving you a boost if you’re creating low-level Linux software. Libbpf generates smaller files that use less memory than those generated by BCC. It also removes dependencies and makes your code simpler, removing the need for kernel imports. That means you can write cleaner code with fewer includes at the start of files, and your output is easier for clients to install.
In a direct comparison between the two, libbpf is a clear winner. Switching from BCC is easy to do. Libbpf’s lower resource use and greater portability simplify your work and make your final product more attractive to your customers, so you have plenty to gain from making the change.
If you're looking to build internal tools to help monitor your applications, consider Airplane. With Airplane, you can create custom internal workflows and UIs to support your engineering workflows. If you need a frontend, Airplane comes with a React-based platform called Views that helps you visualize and monitor your applications in a single interface.
To build your first internal UI using Airplane Views, sign up for a free account or book a demo.
