I believe that reasoning about a program, and knowing where to insert “probes”, is a key part of development that can make the difference between “stuck all afternoon” and finishing a task on time.
This is why I love the idea of BPF: you can instrument any part of your computer. Any function in your program, any syscall, the filesystem, virtual memory, network, you name it. It should really level up my debugging, and is why I’m reading BPF Performance Tools.
BCC is a compiler frontend for BPF, and comes with its own set of observability tools, many of which are documented in the book.
This post is half tutorial, half debugging story; it explains how to Get Started with BCC on Ubuntu 20.04, while recounting the missteps (and how I debugged them) I made along the way.
Installing BCC Tools
Kernel Prereqs
Your kernel needs to have been compiled with the ability for BPF; there are a lot of kernel flags you need to have. I believe the stock Ubuntu kernel has them all, but you should check you have them with this:
bootconfig="/boot/config-$(uname -r)"
while read -r flag;
do
if ! grep -q "$flag" $bootconfig; then
echo "Kernel flag \"$flag\" is missing in $bootconfig"
fi
done << __EOM
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_UPROBE_EVENTS=y
CONFIG_KPROBE_EVENTS=y
__EOM
(Email me if I’ve missed some!)
Installing
I’ll basically link to the thorough INSTALL.md of the repo, as I would basically be copying from there anyways.
Of note is that at the moment the iovisor doesn’t have a release for focal, so you have to use bionic where you see $(lsb_release -cs)
I managed to install from both apt and source without a hitch.
Running BPF Programs
The first BPF program to run as an example is opensnoop. It shows you which programs are opening which files on your computer.
Unfortunately, opensnoop won’t work out of the box:
$sudo opensnoop-bpfcc
create_probe_event: open(/sys/kernel/debug/tracing/kprobe_events): No such file or directory
...
Exception: Failed to attach BPF program b'trace_entry' to kprobe b'do_sys_open'
I had a bit of trouble with this. I had already checked all the kernel flags. I looked into
/sys/kernel/debug/tracing, but it was empty.
It turns out this folder /sys/kernel/debug is a module called debugfs. Some stackoverflow questions are about this folder being empty
because they have disabled debugfs, but you can check you have it mounted with
$ mount | grep sys/kernel
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
So it was already mounted, just not showing tracing events.
Opensnoop works by attaching a kprobe (kernel probe) to the kernel open family of syscalls. You can list which kprobes you have
attached with sudo perf probe -l, but this also errors:
$ sudo perf probe -l
Please rebuild kernel with CONFIG_KPROBE_EVENTS or/and CONFIG_UPROBE_EVENTS.
Error: Failed to show event list.
This isn’t informative, we already have those kernel flags enabled!
The real clue comes out of the dmesg output after trying opensnoop:
[73578.738906] Lockdown: opensnoop: unsafe use of perf is restricted; see man kernel_lockdown.7
[73578.738914] Lockdown: opensnoop: use of kprobes is restricted; see man kernel_lockdown.7
Kernel Lockdown is a recent development in the linux kernel that enables many security features, and complementarily disables observability features like loading kernel modules. If someone could load kernel modules at will, they could really tamper with the system!
Starting with 20.04 Ubuntu has enabled Kernel Lockdown by default:
Starting with Ubuntu 20.04, the Linux kernel's lockdown mode is enabled in integrity mode.
This prevents the root account from loading arbitrary modules or BPF programs that can manipulate kernel datastructures. Lockdown enforcement is tied to UEFI secure boot.
Disabling Kernel Lockdown by disabling Secureboot
As mentioned in that quote snippet, Kernel Lockdown is tied to UEFI Secureboot. The reasoning behind that is that Kernel lockdown is needed to maintain the integrity of the kernel: If you use Secureboot, you can be sure that you are booting a particular kernel, but that doesn’t matter much if someone can tamper with it right after boot.
Warning: This is a fairly intense move. Among other side effects, it might mess with other operating systems you have installed if you dual boot. I will probably re-enable Secureboot after playing with BPF on my personal machine.
You can disable Secureboot with this invocation:
sudo mokutil --disable-validation
mokutil will ask you for a password, and then you need to reboot and type that password in.
The prompt is a bit odd because it repeatedly asks for the nth letter of your password, for a few random n, but w/ever. Pointer: At the end the prompt simply asks “Yes/No” without further clarification; if you want to disable secureboot you need to choose “Yes”.
After booting back into Ubuntu, opensnoop runs just fine:
$sudo opensnoop-bpfcc
PID COMM FD ERR PATH
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
3661 DNS Res~ver #17 124 0 /etc/hosts
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
3661 Cache2 I/O 127 0 /home/amedee/.cache/mozilla/firefox/f1bd6ks0.default-release/cache2/entries/34641B9D8C478316AC8322FD376F9E95DF09FA80
3661 Cache2 I/O 141 0 /proc/self/mountinfo
3661 Cache2 I/O 141 0 /proc/self/mountinfo
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
1098 redis-server 8 0 /proc/1098/stat
932 irqbalance 6 0 /proc/interrupts
932 irqbalance 6 0 /proc/stat
932 irqbalance -1 2 /proc/irq/128/smp_affinity
932 irqbalance 6 0 /proc/irq/160/smp_affinity
932 irqbalance 6 0 /proc/irq/161/smp_affinity
932 irqbalance 6 0 /proc/irq/162/smp_affinity
932 irqbalance 6 0 /proc/irq/178/smp_affinity
932 irqbalance -1 2 /proc/irq/136/smp_affinity
932 irqbalance -1 2 /proc/irq/144/smp_affinity
932 irqbalance 6 0 /proc/irq/134/smp_affinity
932 irqbalance -1 2 /proc/irq/142/smp_affinity
932 irqbalance -1 2 /proc/irq/140/smp_affinity
932 irqbalance -1 2 /proc/irq/139/smp_affinity
This is already interesting:
- I didn’t know I had a Redis server running on my computer, (seems to run on login?)
irqbalanceis a piece of Linux that distributes interrupts across different cores on the computer. I believe it’s either changing or reading the processor affinity of different processes by editing/proc/irq/pid/smp_affinity.- You can see my DNS resolver reading
/etc/hosts, which is cool to see in action!
So there you have it, BPF tools on Ubuntu 20.04 by disabling UEFI Secureboot. Happy hacking!
comments powered by Disqus