Using bpftrace and BCC Tools on CentOS Stream 9: Performance Debugging with eBPF - ITFROMZERO - Share tobe shared! bpftrace and BCC tools are the most powerful observability tools for debugging Linux performance at the kernel level, from I/O latency spikes to unexpected process activity. This guide covers installation on CentOS Stream 9 and practical usage of tools like biolatency, execsnoop, and custom bpftrace scripts.

My production server started showing strange symptoms: normal load average, low CPU, but response time spiking every morning at 7 AM. I stared at top, htop, iostat -x 1 for hours without finding the cause. Traditional tools only give you a snapshot — they don’t tell you why.

Some context: I was managing a fleet of CentOS Stream 9 servers after a rushed migration from CentOS 8 at EOL — at the same time I was migrating 5 servers to Rocky Linux in parallel, which took exactly one week. That experience made me realize that vmstat and sar can show clean numbers while the system is still sluggish — not because the tools are bad, but because they only measure aggregates. bpftrace and BCC are what actually show you what each syscall and each I/O request is doing.

Table of Contents

eBPF, BCC, and bpftrace — What’s the Difference?

These three are often used interchangeably. Getting them straight from the start will save you a lot of time reading documentation.

eBPF (extended Berkeley Packet Filter) is a technology that lets you run code inside the Linux kernel without writing a kernel module. More concretely: you can instrument anything in the system — syscalls, network packets, disk I/O, function calls — with extremely low overhead. The eBPF verifier checks your code before loading it into the kernel, ensuring it won’t crash the system.

BCC (BPF Compiler Collection) is a framework and collection of over 100 pre-built tools on top of eBPF. Each tool solves a specific problem: biolatency for disk I/O, execsnoop for process execution, tcpconnect for network connections. You don’t need to know eBPF internals to use BCC tools.

bpftrace is a high-level tracing language with awk/DTrace-like syntax. Use it when you need to customize something BCC doesn’t cover, or when you want to write a quick one-liner trace.

Simple rule: need a ready-made tool → BCC. Need custom → bpftrace.

Installing on CentOS Stream 9

# Install BCC tools and Python bindings
sudo dnf install -y bcc bcc-tools python3-bcc

# Install bpftrace
sudo dnf install -y bpftrace

# Kernel headers — needed if the kernel doesn't have BTF
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)

# Check BTF support (kernel >= 5.8 usually has it)
ls /sys/kernel/btf/vmlinux && echo "BTF available" || echo "Need kernel headers"

If /sys/kernel/btf/vmlinux exists, you don’t need kernel headers — BTF-based tools will work right away. CentOS Stream 9 with kernel 5.14+ usually has it already.

# Verify installation succeeded
bpftrace --version
ls /usr/share/bcc/tools/ | head -20

Hands-on: Debugging Common Performance Issues

Analyzing I/O Latency with biolatency

This is the first thing I run when I suspect the disk is the culprit. biolatency draws a histogram showing latency distribution in microseconds:

# Monitor disk sda for 10 seconds
sudo /usr/share/bcc/tools/biolatency -d sda 10

# Analyze I/O per process
sudo /usr/share/bcc/tools/biosnoop -d sda

biolatency output looks like this:

     usecs               : count     distribution
         0 -> 1          : 1203     |********                        |
         2 -> 3          : 4521     |********************************|
         4 -> 7          : 2109     |***************                 |
         8 -> 15         : 892      |*******                         |
        16 -> 31         : 45       |                                |
     32768 -> 65535      : 12       |                                | <-- abnormal spike

That last bucket — 12 requests taking 32–65ms — is a red flag. Running biosnoop will tell you exactly which process is responsible.

Tracking Process Execution with execsnoop

Mysterious processes forking? execsnoop captures every exec() syscall in real time:

sudo /usr/share/bcc/tools/execsnoop

Sample output:

PCOMM            PID    PPID   RET ARGS
bash             12345  12344    0 /bin/bash -c /usr/local/bin/check-disk.sh
python3          12346  12345    0 python3 /usr/local/bin/check-disk.sh

This is exactly how I tracked down the morning latency spike: an old backup script running every minute, forking 15–20 subprocesses, and competing for I/O with the web server. Nobody remembered the script still existed.

Debugging File Open Calls with opensnoop

# View all opened files (with return codes)
sudo /usr/share/bcc/tools/opensnoop

# Only monitor the nginx process
sudo /usr/share/bcc/tools/opensnoop -n nginx

# Filter by specific PID
sudo /usr/share/bcc/tools/opensnoop -p 1234

Great for debugging “file not found” errors at the app level, or catching an app reading config from an unexpected path — something that commonly happens after migration when old paths are still hardcoded in config files.

Monitoring TCP Connections with tcpconnect

# View outbound TCP connections
sudo /usr/share/bcc/tools/tcpconnect

# Filter only database and cache ports
sudo /usr/share/bcc/tools/tcpconnect -P 5432,6379,3306

I’ve seen this exact case: after a database migration, the app was silently connecting to the old IP — no obvious errors because the connection timeout was slow, just inexplicably rising latency. tcpconnect exposed it within the first 10 seconds.

CPU Profiling Without Restarting Processes

# Sample CPU stack traces for 30 seconds at 99 Hz
sudo /usr/share/bcc/tools/profile -F 99 30 > /tmp/profile.out

# Profile a specific PID
sudo /usr/share/bcc/tools/profile -F 99 -p 1234 30

Output is a folded stack trace, compatible with flamegraph.pl for visualization. This is the most effective way to debug CPU bottlenecks without recompiling the binary with debug symbols.

bpftrace: Custom Tracing on Demand

Common One-liners

# Count syscalls by name, print on Ctrl+C
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }'

# Measure read() syscall latency — nanosecond histogram
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_read  /@start[tid]/ {
    @latency_ns = hist(nsecs - @start[tid]);
    delete(@start[tid]);
  }'

# Top 10 most frequently opened files
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_openat { @[str(args->filename)] = count(); }
  END { print(@, 10); }'

Script to Trace Slow Disk Writes

#!/usr/bin/env bpftrace
// Run: sudo bpftrace slow_writes.bt
// Report ext4 writes taking > 10ms

kprobe:ext4_file_write_iter
{
    @start[tid] = nsecs;
}

kretprobe:ext4_file_write_iter
/@start[tid]/
{
    $dur_ms = (nsecs - @start[tid]) / 1000000;
    if ($dur_ms > 10) {
        printf("[SLOW WRITE] PID=%d COMM=%s duration=%dms\n",
               pid, comm, $dur_ms);
    }
    delete(@start[tid]);
}

Tracing Function Latency with kprobe

# Measure VFS read operation latency
sudo bpftrace -e '
  kprobe:vfs_read    { @start[tid] = nsecs; }
  kretprobe:vfs_read /@start[tid]/ {
    @us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }'

Notes for Production Use

Overhead: eBPF overhead is very low but not zero. Avoid continuously tracing hot paths (every packet, every syscall) — use intervals or filter by PID/process.
Compile delay: BCC tools compile the BPF program at startup, which can take 1–5 seconds. This is normal, not an error.
Permissions: Root or CAP_BPF + CAP_PERFMON capabilities are required.
Kernel version: Some features require kernel >= 5.8. CentOS Stream 9 qualifies with kernel 5.14+.

In practice, I haven’t seen any significant overhead running these BCC tools for 15–30 seconds to debug and then stopping. Problems only arise if you leave certain tools running continuously for hours on a heavily loaded system without filtering.

Conclusion

The same type of problem used to take 2–3 days to debug. With bpftrace and BCC, I can pinpoint it in minutes — not because the tools are magic, but because instead of guessing from aggregate metrics, you’re looking directly at what the kernel is doing.

The most practical way to start: run biolatency and execsnoop on a test server, look at real output, then find the tool that fits your specific problem. BCC first, bpftrace later — bpftrace syntax is far simpler than writing eBPF C directly, and once you have a BCC foundation you’ll pick it up quickly. Best resources: the bpftrace reference guide on GitHub and Brendan Gregg’s BPF Performance Tools — he’s the author of most of the BCC tools you just used.