Configuring kdump on Fedora Server: Capturing Kernel Panics and Analyzing Crash Dumps - ITFROMZERO - Share tobe shared! A step-by-step guide to configuring kdump on Fedora Server, from reserving crashkernel memory and optimizing dump size with makedumpfile to analyzing vmcore dumps using crash.

Table of Contents

Quick Start: Set Up kdump in 5 Minutes

If you need to immediately safeguard a production server against unexplained system crashes, follow these three steps.

Step 1: Install the kexec-tools Package

Open your terminal and install the kernel image loader package:

sudo dnf install -y kexec-tools

Step 2: Reserve RAM for the Crash Kernel

The rescue kernel needs an isolated memory region to operate when the primary system crashes. Use grubby to configure the boot parameter:

sudo grubby --update-kernel=ALL --args="crashkernel=512M"

Reboot the machine so GRUB applies this memory reservation:

sudo reboot

Step 3: Enable the Service and Verify

Once the system boots back up, enable and start the kdump daemon:

sudo systemctl enable --now kdump.service
sudo kdumpctl status

If the output shows kdump: Kdump is operational, your system is fully prepared to capture crashes at any moment.

How Does kdump Work During a Kernel Panic?

When a kernel panic occurs, all CPUs immediately halt regular task execution. System RAM freezes in place. Traditional logging facilities like rsyslog and systemd-journald become completely unresponsive. After rebooting, you will find zero clues in /var/log/messages.

This is precisely why kdump exists. It functions as an independent flight recorder. Leveraging the kexec mechanism, kdump instantly boots a lightweight capture kernel inside the pre-reserved 512MB memory block. This entire handoff completely bypasses time-consuming BIOS/UEFI hardware re-initialization.

The capture kernel’s task is simple: dump the old kernel’s physical RAM contents, compress it into a vmcore file, and write it to disk. Once completed, the system automatically reboots.

In production environments, a Fedora server running compute-intensive workloads with out-of-tree network drivers can easily suffer from subtle memory corruption or hidden deadlocks. Without a vmcore, you are left blindly guessing through hundreds of git commits or replacing hardware in vain.

Fine-Tuning kdump.conf

The main configuration file is located at /etc/kdump.conf. By default, the system saves dumps to the local disk, but you can tailor this to suit your infrastructure.

1. Change the Dump Storage Location

Writing crash dumps to the root partition (/) poses a risk of filling up disk space. You can redirect the storage path to a dedicated data partition or a remote NFS server:

# Write to a dedicated partition mounted at /mnt/crash
path /mnt/crash

# Or send to an NFS server over the local network
# nfs 192.168.1.50:/data/kdumps

2. Reduce Dump Size Using makedumpfile

For servers with 64GB or 128GB of RAM, generating raw memory dumps consumes substantial disk space and can take tens of minutes. The following configuration strips out unnecessary data before compression:

core_collector makedumpfile -l --message-level 7 -d 31

The -d 31 dump level combines all filtering flags: excluding zero pages, cache memory, user-space application memory, and free pages. The resulting dump retains only kernel data structures. This shrinks file sizes from 64GB down to roughly 600MB–1.2GB.

Hands-On: Triggering a Controlled Panic and Analyzing the Crash

Never wait for a real production outage to discover that your kdump setup has insufficient memory. Proactively trigger a kernel panic in a staging environment.

Simulating a Kernel Panic via SysRq

The Linux kernel supports direct hardware debugging through Magic SysRq. Execute these two commands (note: the system will crash and reboot immediately):

echo 1 | sudo tee /proc/sys/kernel/sysrq
echo c | sudo tee /proc/sysrq-trigger

Once the machine finishes rebooting, inspect the crash directory:

ls -lh /var/crash/

You will see a directory named in the format 127.0.0.1-YYYY-MM-DD-HH:MM:SS/ containing the vmcore binary and a text log summarizing the dmesg output at the moment of the crash.

Analyzing vmcore with the crash Utility

To inspect this binary dump, install the crash tool alongside debugging symbols (debuginfo) that match the exact running kernel version:

sudo dnf install -y crash dnf-plugins-core
sudo dnf debuginfo-install -y kernel-$(uname -r)

Launch the crash analysis utility:

crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/127.0.0.1-*/vmcore

Once the crash> prompt appears, run the bt (backtrace) command to inspect the call stack:

crash> bt
PID: 1420   TASK: ffff88012a45c000  CPU: 1   COMMAND: "tee"
 #0 [ffffc90001423d60] machine_kexec at ffffffff8106a3f3
 #1 [ffffc90001423dc0] __crash_kexec at ffffffff8116248d
 #2 [ffffc90001423e80] panic at ffffffff810be512
 #3 [ffffc90001423f00] sysrq_handle_crash at ffffffff81580125
 #4 [ffffc90001423f10] __handle_sysrq at ffffffff81580872

The call stack clearly reveals the trail: the tee process (PID 1420) invoked the trigger function sysrq_handle_crash. In a genuine crash scenario, this stack trace will point directly to the culpable third-party driver or the kernel function encountering a null-pointer dereference.

Practical Operational Notes

Allocating crashkernel RAM: 512M works well for servers with 8GB to 32GB of RAM. For large servers exceeding 128GB of RAM with multiple RAID controllers or 10GbE/25GbE optical NICs that require large drivers during boot, increase this allocation to 768M or 1024M to prevent the capture kernel from running into Out-Of-Memory (OOM) errors.
Automated Cleanup: Each vmcore file can occupy 1GB–2GB of storage. Multiple consecutive crashes can rapidly exhaust space on /var. Configure systemd-tmpfiles or a cron job to automatically purge crash directories older than 14 days.
Kernel Updates: Whenever you run dnf upgrade to install a newer kernel version, verify with sudo kdumpctl status to ensure dracut has rebuilt a compatible capture initramfs for the new kernel.