Monitoring Memcached with Prometheus & Grafana: Decoding the ‘Blind Spots’ Overloading Your Database – ITFROMZERO

Table of Contents

2 AM and the ‘Slow Query’ Nightmare

My phone vibrated violently on the desk. System alerts were pouring in: Database CPU hit 98%, latency spiked from 50ms to 3000ms. I jumped up to check the logs, half-asleep. Traffic hadn’t surged, yet the primary database was struggling under thousands of queries per second.

The first question that popped up: What on earth is Memcached doing?

Normally, Memcached runs smoothly, handling up to 90% of read traffic. But when an incident occurs, without concrete metrics, you’re essentially flying blind. SSHing into a server and running stats via telnet only gives you a snapshot. It doesn’t show trends or the exact moment things started going south.

That’s why the Prometheus + Memcached Exporter + Grafana combo is a lifesaver. In the systems I operate, this setup helped the team detect full RAM leading to a spike in Evictions before users even started complaining about a slow website.

Why is Memcached Running but the System Still Sluggish?

There are 3 common scenarios where Memcached becomes ineffective that you should watch out for:

Low Cache Hit Rate: Data isn’t in the cache, forcing every request to hit the database directly.
High Evictions: Memcached is out of RAM. It’s forced to delete old keys (even if they haven’t expired) to make room for new data.
Connection Limit: The number of connections from the application has reached its limit, usually 1024 by default.

To avoid pulling all-nighters, you need a visual dashboard to identify exactly where the fault lies at a glance.

Implementing Monitoring with Memcached Exporter

Instead of manual commands, we use Memcached Exporter. This is a lightweight sidecar written in Go. It connects to Memcached, fetches stats, and converts them into a format Prometheus can read.

Step 1: Installing Memcached Exporter

With Docker, you only need a single command to get it running:

docker run -d \
  --name=memcached-exporter \
  -p 9150:9150 \
  prom/memcached-exporter:v0.13.0 \
  --memcached.address=172.17.0.1:11211

Note: Replace the IP 172.17.0.1 with the actual address of your Memcached server.

If you’re using Linux (Binary), download the latest release from GitHub, extract it, and run it as a systemd service for stability:

wget https://github.com/prometheus/memcached_exporter/releases/download/v0.13.0/memcached_exporter-0.13.0.linux-amd64.tar.gz
tar xvf memcached_exporter-0.13.0.linux-amd64.tar.gz
./memcached_exporter --memcached.address="localhost:11211"

Quickly verify by visiting http://localhost:9150/metrics. If you see memcached_up 1, you’re halfway there.

Step 2: Configuring the Scrape Job in Prometheus

Open your prometheus.yml file and define the endpoint for Prometheus to periodically pull data:

scrape_configs:
  - job_name: 'memcached_prod'
    static_configs:
      - targets: ['<EXPORTER_IP>:9150']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'memcached-01'

After reloading Prometheus, data will begin flowing into the Time Series Database.

Step 3: Setting Up the Grafana Dashboard

Don’t waste time building charts from scratch. The community has already optimized professional templates. Simply go to Grafana, select Import, and enter ID 74 or 3932. Instantly, you’ll have charts for Hit Rate, Memory, and Network Traffic.

3 “Golden” Metrics to Monitor Closely

When looking at the Dashboard, ignore the secondary metrics and focus on these 3 critical indicators:

1. Cache Hit Rate

Formula: get_hits / (get_hits + get_misses).

A healthy system typically has a Hit Rate above 90%. If this number drops below 75%, it’s a red flag. This could be due to inconsistent cache key logic in the code or a TTL (Time To Live) that’s too short, causing data to be evicted before it can be reused.

2. Evictions (Items pushed out)

Ideally, Evictions should be 0. If this chart starts showing vertical spikes, it means you’ve run out of RAM. Memcached is being forced to “sacrifice” old data. The solution is to either increase RAM or filter out unnecessary junk objects from the cache.

3. Memory Usage

Don’t let Memcached exceed 80% of its allocated RAM. Memcached’s Slab Allocation mechanism can sometimes make it seem like there’s free memory when, in reality, it can’t store new objects. Set an alert when Memory hits 85% so you can plan a scale-up early.

Real-world Experience: Don’t Wait for an Incident to Check the Dashboard

A beautiful dashboard is just for show if you don’t have an alerting system. I typically set up Telegram alerts with specific thresholds:

Critical: memcached_up == 0 (Server down, immediate action required).
Warning: rate(memcached_items_evicted_total[1m]) > 10 (Clear sign of memory shortage).
Warning: Hit Rate < 70% for 10 consecutive minutes.

Thanks to these rules, I once caught a buggy deployment that changed the key structure, causing the Hit Rate to plummet. The team rolled back within 5 minutes, avoiding a major downtime incident.

Conclusion

Monitoring Memcached isn’t just about installing tools for the sake of it; it’s about understanding how your system ” breathes” through metrics. Combining Prometheus and Grafana makes you proactive rather than reactive. May you have peaceful nights, free from the sound of blaring alerts!