A Guide to bpftrace and BCC on Fedora: Linux Performance Analysis and Diagnostics with eBPF - ITFROMZERO - Share tobe shared! Server lagging while top and iotop report normal metrics? Tap into eBPF with BCC and bpftrace on Fedora to dissect latency, disk I/O, and networking with sub-1.5% overhead.

Table of Contents

When top and htop Fail to Solve the Incident

It is a scenario familiar to almost any systems engineer: Slack is firing red alerts, API p99 latency spikes to 2,500ms, and CPU iowait hits 70%. You quickly SSH into the node and run top, htop, then iotop. But bafflingly, the metrics look completely harmless. The Postgres process sits at a modest 12% CPU usage, and disk I/O hovers around an ordinary 25 MB/s. You are left entirely in the dark: which PID, query, or specific inode is dragging the system down?

The reflexive move during an impasse is to whip out strace to inspect system calls. A few months ago, a junior engineer on my team ran this exact command directly against a production database sustaining 15,000 QPS:

# NEVER run this command on a busy production server!
sudo strace -p $(pgrep -f postgres) -T -tt

The result? The database dropped connections in waves within three seconds. The underlying ptrace mechanism forces the kernel to pause the thread at every syscall entry and exit. When an application processes tens of thousands of I/O operations per second, strace overhead degrades throughput by 40x to 80x. You have just turned an intermittent issue into a catastrophic total outage.

The Fatal Blind Spot Between User Space and Kernel Space

Classic observability tools fall short because of inherent architectural limitations:

Coarse-grained resolution: Tools like top, vmstat, and sar read aggregated metrics periodically from /proc and /sys. If a micro-burst occurs—where an I/O bottleneck chokes the system for 15ms and clears up—a 1-second sampling interval smooths the anomaly away. The charts look flat and serene, but users took the full brunt of the timeouts.
Loss of context: iostat may report that NVMe drive /dev/nvme0n1 is 98% busy, but it cannot pinpoint which thread issued the requests, which file descriptor is stalled, or which block offset is involved.
Intrusive instrumentation: Traditional tracing introduces excessive context switching and latency overhead, rendering it unsafe for production workloads under load.

This is precisely the challenge that eBPF (Extended Berkeley Packet Filter) solves cleanly. Fedora provides an ideal proving ground because it consistently ships with the latest upstream Linux kernels. Instead of copying data back and forth between kernel and user space, eBPF loads bytecode verified safe by the in-kernel verifier directly into the operating system kernel. By attaching straight to kprobes, tracepoints, or uprobes, measurement overhead typically stays well below 1.5%.

Three Approaches to eBPF on Fedora: From Off-the-Shelf to Tailor-Made

Depending on time pressure and incident severity, you have three primary approaches:

Method 1: Leverage Ready-to-Use BCC (BPF Compiler Collection) Tools

BCC bundles dozens of battle-tested Python and C scripts curated by Brendan Gregg and the performance community. You can resolve 80% of common production incidents without writing a single line of code.

Install the package suite on Fedora:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r) kernel-headers-$(uname -r)

Field tip: All executables reside in /usr/share/bcc/tools/ rather than /usr/bin to prevent naming collisions. Add this directory to your PATH or invoke scripts using absolute paths.

Catch disk bottlenecks red-handed with biolatency-bpfcc:

# Measure block I/O latency in milliseconds and summarize into a histogram
sudo /usr/share/bcc/tools/biolatency-bpfcc -m 5

Instead of relying on misleading averages, you see the true latency distribution immediately: how many I/O operations completed under 1ms, and how many requests backed up into the dangerous >512ms tier due to saturated storage queues.

Catch short-lived processes with execsnoop-bpfcc:

# Instantly trace all newly forked/exec'd processes across the system
sudo /usr/share/bcc/tools/execsnoop-bpfcc -T

This tool is invaluable when CPU spikes to 100% momentarily and disappears before top completes its 1-second refresh cycle—often caused by short-lived cron subshells spawning and exiting immediately.

Method 2: Build Native eBPF Programs with C and libbpf (CO-RE)

This approach uses pure C compiled via Clang into lean ELF bytecode binaries loaded by libbpf. It represents the gold standard for long-running observability daemons like Datadog or Cilium due to minimal memory footprints and zero requirement for heavy toolchains on production nodes.

The trade-off is development lead time. You must manually manage BPF ring buffers and maps while writing custom user-space ingestion logic. During an active fire drill, you cannot afford hours writing and debugging C code just to inspect a single kernel function.

Method 3: Rapid Diagnosis with bpftrace One-Liners

If BCC is a preassembled wrench set and libbpf is a full-scale machine shop, bpftrace is the Swiss Army knife. Its hybrid awk-and-C syntax lets you write powerful, expressive one-liners straight from the command line.

Install the package on Fedora:

sudo dnf install -y bpftrace

Audit which processes are opening files on the fly:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("[%s] (PID %d) opened: %s\n", comm, pid, str(args.filename)); }'

Rank top disk-reading processes every 5 seconds:

sudo bpftrace -e 'kprobe:vfs_read { @[comm] = count(); } interval:s:5 { print(@); clear(@); }'

With a single concise command, you intercept calls to vfs_read, aggregate them by process name, and print summary tables periodically. Fast, precise, and negligible resource usage.

Battle-Tested Triage Workflow: Combining BCC and bpftrace

After navigating countless late-night production war rooms, I have refined this two-stage diagnosis playbook:

Stage 1 – Macro Isolation with BCC:

Do not jump into custom scripting before isolating the bottleneck. Start with standard exploratory tools:
- Suspecting I/O: Run biolatency-bpfcc to inspect latency profiles and biosnoop-bpfcc to identify culprit PIDs.
- Suspecting Network: Run tcpconnect-bpfcc or tcptracer-bpfcc to evaluate TCP handshake latencies.
- Suspecting Memory: Execute memleak-bpfcc -p <PID> to track down unreleased malloc allocations.
Stage 2 – Microscopic Inspection with bpftrace:

Once BCC flags a target PID or subsystem, point bpftrace directly at the problem area like a microscope. For instance, if an Nginx worker appears stalled on disk cache reads, trace the latency distribution of the read syscall specifically for that process:

# Measure read syscall latency specifically for nginx processes (in microseconds)
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_read /comm == "nginx"/ {
    @start[tid] = nsecs;
  }
  tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @duration_us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }
'

Three Critical Hard-Won Lessons on Fedora

Verify Kernel Headers Alignment: Fedora releases regular kernel updates. If you recently executed dnf upgrade without rebooting, the newly installed kernel-devel package will not match the active running kernel (uname -r). Both BCC and bpftrace will fail to compile scripts immediately. Always cross-check rpm -q kernel-devel against uname -r before diving into diagnostics.
Account for SELinux Enforcement: Fedora runs SELinux in Enforcing mode by default. Even when executing as root or with CAP_BPF, SELinux policy may block tracing hooks on restricted daemons. If your commands fail with unexpected permission denied errors, inspect journalctl -t setroubleshoot or check /var/log/audit/audit.log.
Explicitly Free Tracking Maps in bpftrace: In stateful scripts, maps like @start[tid] allocate non-swappable kernel RAM. When tracing high-frequency code paths without cleaning up via delete(@start[tid]) after calculating latency, kernel maps can grow unbounded and trigger out-of-memory conditions.