Monitoring Every Linux Command: Snoopy Logger and Grafana Loki Solution – ITFROMZERO

Table of Contents

Why .bash_history Isn’t Enough for Monitoring

Early in my career, I learned a painful lesson. A database server crashed because someone accidentally modified a config file. I excitedly SSH’ed in and typed history to find the culprit, but the result was a blank screen.

It turned out the person responsible had quickly typed history -c to cover their tracks. Even without that, when multiple sessions are used simultaneously, default Linux logs can easily be overwritten, leading to data loss. Default command history is extremely fragile.

If you manage a system with 5-10 users, you need a lower-level logging solution. It must capture every command even if the user tries to hide it. That’s why I chose Snoopy Logger combined with Grafana Loki.

Snoopy Logger – A “Dashcam” for the Terminal

Snoopy Logger isn’t a heavy shell script or background software. Essentially, it’s a shared library that intercepts the system’s execve() function via the LD_PRELOAD mechanism.

Every time a new process starts—from ls and rm to background scripts—Snoopy immediately captures it and pushes it directly to syslog. The advantages that keep me using it include:

Detailed recording: User ID (UID), Terminal (TTY), Current Working Directory (CWD), and the original command.
Tamper-proof: Users generally cannot disable or bypass it by clearing their history.
Impressive performance: Consumes less than 0.1% CPU even when the system is processing thousands of commands per minute.

Centralized Deployment Model

Previously, I had to SSH into each server and run tail -f /var/log/auth.log to troubleshoot. This was too manual and time-consuming. Now, I forward all logs to Grafana Loki via Promtail.

Loki provides excellent log compression, saving up to 80% storage compared to plain text. When an investigation is needed, I just open the Grafana Dashboard, filter by server name or username, and every action becomes crystal clear.

Detailed Implementation Guide

Step 1: Install Snoopy Logger

On Ubuntu or Debian, you only need a single command:

sudo apt update && sudo apt install snoopy -y

To check if the “camera” is recording, try running a few commands and check the system logs:

# Try executing a command
cat /etc/passwd

# Check logs immediately
sudo tail -f /var/log/auth.log | grep snoopy

The resulting log line will contain full identification information such as the UID of the user and the exact command executed.

Step 2: Configure Promtail to Transport Logs

Snoopy logs locally, while Promtail acts as the shipper to send logs to the Loki server. Create the configuration file /etc/promtail/config-snoopy.yaml with the following content:

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://<LOKI_SERVER_IP>:3100/loki/api/v1/push

scrape_configs:
- job_name: snoopy_monitoring
  static_configs:
  - targets:
      - localhost
    labels:
      job: snoopy_logs
      host: prod-web-01
      __path__: /var/log/auth.log

Start Promtail to begin pushing data to the monitoring center.

Step 3: Query and Parse Data on Grafana

In the Grafana Explore interface, select the Loki data source. Use a simple LogQL query to filter Snoopy logs:

{job="snoopy_logs"} |= "snoopy"

In my experience, you should use the pattern function in LogQL. It helps separate the UID, execution directory, and command into individual columns. This makes your Dashboard much more intuitive compared to reading raw text.

Real-world Operational Experience

Real-world deployment for large systems often brings up data noise issues. Here are 3 tips to help you optimize the system:

1. Control Log Volume

By default, Snoopy records everything. If a server runs hundreds of cron jobs per minute, the log file will grow very quickly. You should configure the /etc/snoopy.ini file to filter out unnecessary users.

# Exclude logs from unimportant system users
[snoopy]
filter_chain = "exclude_uid:100,101"

2. Ensure Data Integrity

While Snoopy is powerful, if an attacker gains root access, they can still delete local logs. Pushing logs to Loki in real-time is mandatory. Once logs have left the original server, the attacker has no way to erase the traces recorded on Loki.

3. Set Up Automated Alerts

Don’t wait until an incident occurs to check the logs. I usually set up Alerting on Grafana to receive Telegram notifications as soon as someone executes sensitive commands like rm -rf /, chmod 777 or accesses directories containing secret keys.

Conclusion

Combining Snoopy Logger and Grafana Loki brings absolute transparency to your Linux systems. It not only helps in tracing incidents but also serves as crucial evidence during security audits.

If you are managing critical infrastructure, try implementing this model today. It will save you from sleepless nights wondering “who did what” to your server. Good luck!