Posted inMonitoring

Integrating CrowdSec with Prometheus and Grafana: Linux Server Attack Monitoring and Alerting

Monitoring tutorial - IT technology blog
Monitoring tutorial - IT technology blog

1. When the Server Stumbles but Grafana Stays Green

At 2:00 AM on a weekday, our team’s three-node Nginx and API backend cluster suddenly ground to a halt. I jumped straight into Grafana: CPU usage was under 35%, free RAM was over 60%, and disk I/O was dead flat. Prometheus remained dead silent, without a single hardware overload alert.

Yet customers were flooding support with 504 Gateway Timeout errors. Hurriedly SSHing into the servers to inspect the Nginx logs, I was shocked: over 9,000 requests per minute were hammering the system. We were enduring a barrage of WordPress vulnerability scans and SQL injection attempts originating from more than 400 distributed IPs. At the same time, the SSH port was choking from a brute-force attack.

Fail2ban on the servers was completely overwhelmed. Regex parsing gigabytes of log files caused the Python process to stutter. Worse yet, it had no way to synchronize the malicious IP blocklist across the other two nodes.

Most frustrating of all, the operations team was flying completely blind against application-layer attacks. We lacked real-time visibility: How many malicious IPs are out there? What attack vectors dominate? What is our block rate?

2. The Gap Between System Monitoring and Security Tools

Dissecting the incident revealed three clear structural flaws in our legacy monitoring setup:

  • Node Exporter only sees hardware: Prometheus knows CPU percentage, but it is completely oblivious to 15,000 HTTP 403 responses per minute or 50 consecutive failed SSH login attempts.
  • Fail2ban lacks a native metrics exporter: Fail2ban stores its state in flat log files or local SQLite databases. Scraping that data into Prometheus for graphing requires cumbersome custom scripts.
  • Isolated defense without threat intelligence: Fail2ban only reacts after an attacker has already knocked on your door repeatedly. If an IP has just swept thousands of other servers across the internet, your server still naively welcomes it with open arms.

3. Three Approaches and Their Trade-offs

To bring security data into a unified dashboard, I evaluated three potential approaches:

Approach 1: Writing a Custom Log Parser for Prometheus Textfile Collector

I once ran a Python script on a 1-minute cron job to count ‘Failed password’ lines and dump metrics into a .prom file. While quick to set up, log files ballooning to several gigabytes consumed excessive CPU. Moreover, metrics were prone to data drops whenever logrotate triggered.

Approach 2: Shipping All Logs to an Elasticsearch / Graylog Cluster

Aggregating logs into a centralized SIEM is invaluable for post-incident forensics. However, an ELK stack demands at least 8GB to 16GB of RAM. For small to medium-sized server fleets, this resource overhead is prohibitively expensive. Furthermore, querying logs to trigger alerts always incurs greater latency than Prometheus periodic metric pulls.

Approach 3: Replacing Fail2ban with CrowdSec and Enabling Prometheus Metrics

CrowdSec also analyzes logs to detect malicious activity, but it is written in Go. It is remarkably lightweight, consuming only about 60MB to 80MB of RAM in production. Crucially, CrowdSec comes with a built-in Prometheus endpoint on port 6060. Prometheus can directly scrape metrics on banned IPs, triggered scenarios, and log processing throughput without requiring any external exporters.

4. Deploying CrowdSec with Prometheus and Grafana

Here is the 5-step setup that our team has been running reliably on Ubuntu 22.04 LTS for the past 6 months.

Step 1: Install CrowdSec and the Remediation Component (Bouncer)

Install the CrowdSec engine along with the iptables bouncer to automatically drop packets from malicious IPs:

curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash
sudo apt-get update
sudo apt-get install crowdsec
sudo apt-get install crowdsec-firewall-bouncer-iptables

CrowdSec automatically scans running services (such as Nginx and SSH) and preloads the matching parsers.

Step 2: Enable the Prometheus Metrics Endpoint in CrowdSec

By default, CrowdSec has the Prometheus endpoint enabled, bound to 127.0.0.1:6060. Open the configuration file:

sudo nano /etc/crowdsec/config.yaml

Locate the prometheus block. Leave it as 127.0.0.1 if Prometheus resides on the same host, or update it to your private network IP:

prometheus:
  enabled: true
  listen_addr: 127.0.0.1
  listen_port: 6060

Restart CrowdSec to apply the changes:

sudo systemctl restart crowdsec

Verify that the endpoint exposes metrics:

curl -s http://127.0.0.1:6060/metrics | grep cs_

You should see metrics prefixed with cs_, such as cs_alerts, cs_active_decisions, and cs_reader_hits_total.

Step 3: Add a Scrape Job to Prometheus

Open prometheus.yml on your Prometheus server and define the scrape job:

scrape_configs:
  - job_name: 'crowdsec'
    scrape_interval: 15s
    static_configs:
      - targets: ['10.0.1.15:6060']
        labels:
          instance: 'web-prod-01'

Reload Prometheus configuration without restarting the service:

curl -X POST http://localhost:9090/-/reload

Step 4: Build a Visual Dashboard in Grafana

CrowdSec provides an excellent official Grafana dashboard. Navigate to Grafana > Dashboards > Import, enter dashboard ID 13380 or 14115, and select your configured Prometheus data source.

Key panels to monitor:

  • Active Decisions: The number of currently banned IPs in real time.
  • Alerts by Scenarios: Attack breakdown (e.g., ssh-bf, http-probing, crawl-non-statics).
  • Parser Performance: Log read and parsing rate per second.

Step 5: Configure Alert Rules and Tackle Alert Fatigue

Create an alerting rule file named crowdsec_alerts.yml for Prometheus:

groups:
  - name: CrowdSecSecurityAlerts
    rules:
      - alert: SuddenAttackSpike
        expr: sum(increase(cs_alerts[5m])) by (instance) > 20
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Server {{ $labels.instance }} is under heavy attack"
          description: "More than 20 new violation alerts recorded in the past 5 minutes."

In our early rollout days, we configured Telegram alerts for every single blocked IP. The result: phones buzzed non-stop all day due to mundane automated scans. The team suffered severe alert fatigue and nearly muted the channel altogether.

A hard-learned lesson is to alert on spike velocity rather than isolated events. Set a Warning threshold when more than 20 IPs are banned within a 5-minute window. Escalate to Critical only when an attack is accompanied by hardware saturation or a sudden surge in HTTP 5xx error rates.

Share:
Tags: