When working with Linux, especially managing servers, you will inevitably encounter “tough” issues. A service isn’t running, an application suddenly stops, or the entire system becomes sluggish for no apparent reason. When things don’t go as planned, the first thing we need to do is understand “why.” That’s when powerful logging tools like journalctl and dmesg come into play.
In this article, I will share how to effectively use these two tools for debugging in a Linux environment. The goal is to help you, especially those new to IT, develop a systematic approach when facing server problems.
Comparing Log Reading Methods: Traditional, journalctl, and dmesg
Before diving into each tool, let’s review how log files are managed on Linux, and the reasons behind the emergence of journalctl and dmesg.
Traditional Log Reading Method (/var/log)
When I first started learning Linux, whenever an error occurred, I would often cd /var/log and then use grep to search in files like syslog, auth.log, kern.log, messages… This is a basic approach, but still useful in many cases. These log files are typically managed by rsyslog or syslog-ng, and are plain text files easily readable with cat, less, tail, and grep.
cat /var/log/syslog | grep "error"
tail -f /var/log/nginx/access.log
Pros: Simple, easy to understand, and works well with existing text processing tools.
Cons: Logs are scattered (each service/log type might be in a separate file), lacking a unified structure. This makes efficient filtering and searching difficult, especially on busy systems with massive log volumes.
journalctl: Systemd’s Centralized System Log
With the advent of Systemd – a modern init system and service manager – log management has significantly changed. journalctl is the tool for reading and managing the Systemd Journal, a centralized logging system. Instead of writing to many separate files, Systemd collects logs from the kernel, services, and applications, then writes them to a binary database.
Pros:
- Centralized: All logs from multiple sources are gathered in a single location.
- Structured: Logs are stored in a structured format, making filtering and searching extremely powerful and fast.
- Contextual: A service’s logs always come with full information about that service (PID, Unit name, etc.).
- Boot-specific support: Easily view logs from previous boot sessions.
- Persistence: Can be configured for logs to be permanently stored across reboots.
Cons:
- Logs are stored in a binary format, meaning they cannot be read directly with
catorgrep;journalctlmust be used. - For newcomers, the
journalctlsyntax can be a bit “tricky” with its many options.
dmesg: Messages from the Kernel Ring Buffer
dmesg (display message) is a tool that shows messages from the Kernel Ring Buffer. This is where the operating system kernel records events related to the boot process, hardware identification, driver errors, or other important notifications from the kernel. dmesg logs are often the first information you need to check when encountering hardware issues, driver problems, or errors that occur very early in the system boot process.
Pros:
- Direct from kernel: Provides crucial information about hardware, drivers, and kernel-level errors.
- Available even during severe system failures: Information from the kernel ring buffer is often still accessible even when other services are not yet stable or have failed.
Cons:
- Only displays kernel logs, not including logs from user applications or services.
- Does not offer long-term log history storage by default (ring buffer content is overwritten when full).
- Output can be difficult to read if not filtered or re-formatted.
Choosing the Right Tool: When to Use journalctl, When to Use dmesg?
Choosing the right tool will save you a lot of time when debugging. I usually apply the following rule:
- Start with
journalctl: For most issues related to services (like Nginx web server, PostgreSQL database, SSH), applications, or general system errors, begin your search injournalctl. This is the most centralized and comprehensive log source. - Switch to
dmesgwhen needed: Ifjournalctlshows errors related to the kernel, hardware (e.g., hard drive errors, network cards, RAM), drivers, or if the system experiences issues right from boot-up andjournalctldoesn’t provide enough information, that’s whendmesgwill be effective.
Implementation Guide: Debugging with journalctl and dmesg
Now, we will delve into how to use each tool with practical examples.
Debugging with journalctl
journalctl is an extremely flexible tool. Below are the basic and advanced commands I frequently use:
1. View all system logs
The simplest command to view all logs collected by the Systemd Journal:
journalctl
The results will be displayed in a text viewer (pager) like less, allowing you to scroll freely.
2. View latest logs and follow in real-time
To view the latest logs and continuously follow incoming logs in real-time (similar to tail -f):
journalctl -f
This command is extremely useful when you want to immediately check the impact of a configuration change or after restarting a service.
3. View logs for a specific service
When a service isn’t behaving as expected, you need to view its specific logs. For example, to view Nginx logs:
journalctl -u nginx.service
Or for the SSH service:
journalctl -u ssh.service
You can combine it with -f to follow that service’s logs in real-time: journalctl -u nginx.service -f.
4. Filter logs by time
This is one of journalctl‘s most powerful features. You can filter logs by start time (--since or -S) and end time (--until or -U).
To view logs from 1 hour ago:
journalctl -S "1 hour ago"
To view logs from the beginning of today:
journalctl -S "today"
To view logs within a specific time range:
journalctl -S "2026-03-24 10:00:00" -U "2026-03-24 10:30:00"
5. View logs from previous boot sessions
When encountering issues after a server reboot, you might want to view logs from the previous boot session.
View a list of boot sessions:
journalctl --list-boots
View logs from the previous boot session (-1 is the most recent, -2 is the one before that):
journalctl -b -1
Personal experience: On a production Ubuntu 22.04 server with 4GB RAM that I manage, using journalctl -b -1 -u myapp.service -S "20 minutes ago" when debugging a service error after a reboot significantly reduced my troubleshooting time. Instead of sifting through hundreds, or even thousands, of irrelevant log lines, I could immediately focus on the necessary information for that service within a specific timeframe.
6. Filter by priority level
You can filter logs by priority level, from debug to emergency warnings.
Only display errors:
journalctl -p err
Only display warnings and errors:
journalctl -p warning..err
Priority levels in order from most severe to least severe: 0 (emerg), 1 (alert), 2 (crit), 3 (err), 4 (warning), 5 (notice), 6 (info), 7 (debug).
7. Ensure Persistent Logging
By default, on some Linux distributions (like Ubuntu), the Systemd Journal only stores temporary logs in RAM (volatile). This means logs will be lost after a reboot. To ensure logs are stored permanently, you need to create the directory:
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
After creating this directory, the Systemd Journal will automatically save logs to /var/log/journal, ensuring you can review logs from previous boot sessions without data loss.
Debugging with dmesg
dmesg has a simpler syntax than journalctl, but the information it provides is very unique and specialized.
1. View all kernel messages
The basic command to view the entire content of the kernel ring buffer:
dmesg
Since the output is often very long, you should combine it with less or grep for easier reading:
dmesg | less
dmesg | grep -i "error"
dmesg | grep -i "usb"
2. More readable format with -H
The -H (human-readable) option helps format dmesg output more clearly, with colors and relatively easy-to-understand timestamps. This is the option I use most often.
dmesg -H
When combined with less, you will have a much better kernel log reading experience:
dmesg -H | less
3. View latest messages
To view the latest kernel messages, you can use the following command:
dmesg --follow
This command will continuously display new messages as they are written to the kernel ring buffer, which is very convenient for real-time event monitoring.
4. Check hardware events
When encountering issues with hard drives, network cards, or USB devices, dmesg is the first place you should check. For example, to view messages related to SATA hard drives:
dmesg | grep -i "ata"
Or about USB devices:
dmesg | grep -i "usb"
Conclusion
Mastering journalctl and dmesg is a significant step towards becoming a professional system engineer or DevOps. These two tools, despite their different purposes and operating methods, complement each other very well in diagnosing and troubleshooting issues on Linux. journalctl provides a comprehensive, detailed overview of service and application activity, while dmesg acts as your “eyes and ears” at the kernel level, helping detect potential hardware problems.
Practice regularly with the commands I’ve introduced. You’ll find that debugging on Linux is no longer a daunting challenge, but rather an important skill that helps you manage servers more confidently and effectively. I wish you success!
