Beautiful Dashboards, but the Server is Down and No One Knows
You just spent all week building a monitoring system with Prometheus and Grafana. CPU and RAM charts are dancing professionally on the big screen. However, at 2 AM, the database hangs, causing the website to go down completely. In the morning, you wake up to dozens of missed calls from your boss, while the dashboard shows a flat red line from several hours ago.
No matter how flashy a dashboard is, it’s still just passive monitoring. Without someone watching the screen, all that data is meaningless. What you really need is an active alerting mechanism. The system must know how to “ring the bell” to notify the person in charge as soon as an incident occurs.
Why Do We Often Miss Incidents?
The fault usually lies not in the tools, but in suboptimal operational processes. In my experience, there are three main barriers to successful monitoring:
- Siloed data: Metrics stay within the Prometheus database with no outgoing channel.
- Hesitation to configure SMTP: Many people are wary of touching Grafana’s config files for fear of syntax errors or Gmail security issues.
- Incorrect Thresholds: Alerts that are too sensitive cause noise, while those that are too late mean damage has already been done.
While Telegram or Slack are popular, Email remains an indispensable formal channel. It provides an audit trail for reporting and easily integrates into ticketing systems like Jira or ServiceNow.
3 Steps to Configure Grafana Email Alerting
We will perform this on Grafana v9/v10 using the unified Alerting interface. The process includes: Configuring SMTP, setting up a Contact Point, and creating Alert Rules.
Step 1: Enable SMTP in the Configuration File
Grafana doesn’t send emails natively; it needs a Mail Server. If you’re using Gmail, you must create an App Password instead of using your personal password.
Open the terminal and edit the system configuration file:
sudo nano /etc/grafana/grafana.iniFind the [smtp] section. Remove the semicolon ; at the beginning of each line to enable the following parameters:
[smtp]
enabled = true
host = smtp.gmail.com:587
user = [email protected]
password = your-16-character-app-password
from_address = [email protected]
from_name = Grafana MonitorSave the file and restart the service to apply the changes:
sudo systemctl restart grafana-serverStep 2: Create a Contact Point via the UI
Log in to Grafana, navigate to Alerting > Contact points, and follow these steps:
- Click + Add contact point and give it a recognizable name like
Ops-Team-Email. - In the Integration section, select Email.
- Enter the recipient email addresses, separated by commas.
- Click Test. If your inbox receives a test mail, the system is working correctly.
Step 3: Set Up a Practical Alert Rule
Let’s try creating an alert for when server CPU exceeds 90% for 5 minutes. Go to Alerting > Alert rules:
- Query: Select Prometheus and enter
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100). - Evaluation behavior: Set For to
5m. This helps filter out cases where CPU just spikes for a few seconds and then drops. - Details: Name it “High CPU Usage” and add the label
severity=critical. - Notifications: Select the Contact Point created in Step 2.
Field Experience: Avoiding the Alert Fatigue Trap
A common mistake I used to make was setting alerts for everything. CPU > 70% alerts, RAM > 80% alerts, Disk > 85% alerts. The result? I’d wake up to 500 emails. This is known as Alert Fatigue.
When your inbox is flooded with junk mail, you start ignoring them. By the time a truly critical incident occurs, it gets buried in that pile of trivial notifications. Apply these three golden rules:
- Only alert when action is required: If CPU is high but the system handles it automatically, don’t send an email. Only alert when manual human intervention is needed.
- Grouping: Instead of receiving 20 emails for 20 servers going down at once, configure a Notification Policy to receive 1 email listing all 20 servers.
- Fine-tune thresholds periodically: Application loads change over time. Spend 15 minutes a week reviewing whether your alert thresholds are still appropriate.
Tips for Professional Operations Management
Instead of sending mail to individuals, use an Email Alias or Mailing List like [email protected]. This ensures that when staff leave or shifts change, you don’t have to manually update the configuration in Grafana.
Additionally, use a multi-channel approach. Use Telegram for “Warning” level alerts for quick resolution, and reserve Email for “Critical” levels for archival and auditing. Good luck with your setup, and may you sleep soundly without being startled by unexpected incidents!
