App crashing without a trace?
System Admins and Backend developers are likely familiar with the haunting message: Segmentation fault (core dumped). It’s most frustrating when an application dies suddenly, leaving log files empty because the process was killed too quickly, perhaps by the OOM Killer. In these moments, Core Dump is your only lifesaver.
Think of a Core Dump as an airplane’s “black box.” When an app crashes, Linux takes a snapshot of the entire memory (RAM), registers, and stack, then writes it to a file. Looking at it, you’ll know exactly which line of code caused the error and which variable held the wrong value. From my experience, you should configure Core Dumps as soon as you set up a server; don’t wait until the app crashes in Production to scramble to enable it.
Quick start: Enable Core Dumps in 30 seconds
Typically, Linux distributions like Ubuntu or CentOS set the core file size limit to 0. This means the system won’t generate any files when a crash occurs. To check and quickly enable it for the current session, use these commands:
# Check current limits
ulimit -c
# Allow core file creation with unlimited size
ulimit -c unlimitedTest it now with a Python snippet that triggers an invalid memory access error:
python3 -c "import ctypes; ctypes.string_at(0)"If you see (core dumped) next to the error message, congratulations—the “crime scene” evidence has been recorded.
Advanced and Permanent Core Dump Configuration
The ulimit command only applies temporarily to that specific terminal session. To have the system automatically record every crash after a reboot, you need to modify the system configuration.
1. Permanent configuration with limits.conf
Open the /etc/security/limits.conf file and add the following two lines at the end:
* soft core unlimited
* hard core unlimitedThe * applies to all users. After saving, log out and log back in for the new configuration to take effect.
2. Organizing storage location and file naming
By default, core files are often scattered in the execution directory with the name core, making them hard to manage. I usually gather them into a dedicated directory for easier tracking and cleanup.
# Create a centralized directory
sudo mkdir -p /var/crash
sudo chmod 777 /var/crash
# Configure file name format via sysctl
sudo sysctl -w kernel.core_pattern=/var/crash/core-%e-%p-%tParameter meanings:
%e: Executable filename (e.g., nginx, python).%p: PID of the crashed process.%t: Crash time as a UNIX timestamp.
To maintain this setting after a reboot, add the line kernel.core_pattern=/var/crash/core-%e-%p-%t to the /etc/sysctl.conf file.
Using coredumpctl on Modern Linux Distributions
If you’re using Ubuntu 20.04+, Debian 11+, or RHEL 8+, the system usually comes with systemd-coredump integrated. This tool is incredibly powerful because it manages core files centrally and supports fast querying, similar to how you would be mastering systemd-run for other system tasks.
# List recent crashes
coredumpctl list
# View detailed information of a crash (based on PID)
coredumpctl info 1234
# Extract the core file for separate analysis
coredumpctl dump 1234 -o my_app.coreAnalyzing Errors with GDB
Now that you have the core file, it’s time for the “autopsy.” The most powerful tool for this is gdb (GNU Debugger). While you can learn how to use strace to debug applications on Linux for live tracing, GDB is the standard for post-mortem analysis. Suppose the application my_app crashed; run the following command:
gdb ./my_app /var/crash/core-my_app-1234Type the bt (backtrace) command in the gdb interface. The system will display the list of functions called just before the crash. The culprit is usually found in the last line of code listed.
(gdb) bt
#0 0x0000555555555151 in crash_function () at main.c:10
#1 0x000055555555516a in main () at main.c:15Looking at the results, we know immediately that the error is on line 10 of the main.c file. No more guessing! To take your troubleshooting skills further, you can debug Linux like a “wizard” with eBPF and BCC Tools for real-time observability.
Crucial “Hard-Learned” Notes to Avoid Server Crashes
Core Dump is a double-edged sword. Before deploying it widely, remember these 3 rules:
- Storage Management: Core files are roughly the same size as the RAM the app was using (RSS). If a Java app consuming 16GB of RAM crashes repeatedly, the server’s disk will fill up in minutes. Set quotas or use periodic cleanup scripts.
- Security Risks: Core files contain raw data from RAM. If the app was processing tokens, passwords, or secret keys, they will appear as plain text in this file. Restrict access permissions to the crash directory tightly.
- Debug Symbols: To see function names and line numbers, the app must be built with the
-gflag (debug symbols). Otherwise, you’ll only see cryptic memory addresses like0x00007f...which are very difficult to read.
In summary, configuring Core Dumps helps you shift from a reactive to a proactive stance when handling incidents, much like hunting down CPU-hogging “culprits” on Linux. Try setting it up in your Staging environment today to get familiar with the troubleshooting workflow!
