Real-world Issue: When Containers “Die” in Silence
Two years ago, I was operating a cluster of 20 microservices for an e-commerce platform. One night, the payment feature failed completely. Customers started complaining on the Fanpage, the boss was blowing up my phone, but the monitoring dashboard was still showing all green. As it turned out, a service was hung (zombie process) rather than completely crashed. Docker kept restarting it, but it fell into a CrashLoopBackOff cycle.
It took me over 2 hours to find the cause: a container had a Memory Leak. Every 5 minutes, it would ‘die.’ At that time, I was completely reactive. I only knew about the incident when customers spoke up, instead of knowing as soon as the system started becoming unstable. The lesson learned? Don’t put absolute faith in the --restart always flag, especially when debugging lightweight Docker containers.
Automatic restarts are just a band-aid. What you really need is Observability. You must react instantly to what is happening inside the Docker Engine.
Why Traditional Tools Sometimes Fall Short
Usually, your first thought might be Prometheus + Grafana or the ELK Stack. However, these solutions reveal several drawbacks in small to medium-sized projects compared to an ultra-lightweight Docker logging solution:
- Latency: Prometheus operates on a “pull” mechanism. If you set a 30-second interval, it could take up to half a minute to know a container has died.
- Resource Intensive: Running a full monitoring stack consumes at least 1-2GB of RAM. This is a massive waste if you’re only running 5-10 containers on a low-spec VPS.
- Difficult to Automate: Programming complex response scenarios (e.g., clearing Redis cache before restarting a container) using these tools is often very cumbersome.
We need an Event-driven mechanism. As soon as the Docker Engine performs an action (die, stop, oom), it should fire a signal immediately.
3 Approaches to “Catching” Docker Events
Here are the methods I’ve tried:
- Polling
docker ps: Writing a cronjob script that runs every minute. This is terrible because it wastes resources and has high latency. - Using Log Drivers: Pushing logs to a centralized server. This is good for debugging but extremely difficult for triggering automated actions based on container status.
- Docker Events API: This is the “secret weapon” already available. It provides a real-time data stream of every change in containers, images, and networks.
Solution: Building an Automation System with Docker Events API
Combining the Docker Events API with a lightweight Python script is the most effective approach. This script listens directly to the Docker Socket (/var/run/docker.sock). Then, it pushes alerts via Telegram and executes smart restart logic.
1. Listening to Docker Events via Command Line
You can quickly test this with a bash command to see the returned data:
docker events --filter 'event=die' --filter 'event=oom'When a container is killed or runs out of memory (OOM), detailed information will appear instantly. This is the valuable data source we will exploit.
2. Monitoring and Telegram Alert Script
Below is the Python script structure I use for real-world projects. It’s lightweight (consuming only about 30-50MB of RAM) but extremely powerful.
import docker
import requests
import os
# Telegram Configuration
TELEGRAM_TOKEN = os.getenv("TELEGRAM_TOKEN")
CHAT_ID = os.getenv("CHAT_ID")
def send_telegram_msg(message):
url = f"https://api.telegram.org/bot{TELEGRAM_TOKEN}/sendMessage"
data = {"chat_id": CHAT_ID, "text": message, "parse_mode": "Markdown"}
try:
requests.post(url, data=data, timeout=5)
except Exception as e:
print(f"Error sending Telegram: {e}")
def monitor_events():
client = docker.from_env()
print("🚀 Listening for Docker events...")
for event in client.events(decode=True):
status = event.get('status')
attributes = event.get('Actor', {}).get('Attributes', {})
container_name = attributes.get('name', 'Unknown')
if status == "die":
exit_code = event.get('Actor', {}).get('Attributes', {}).get('exitCode')
msg = f"🔴 *Container Alert*\n*Service:* {container_name}\n*Status:* Stopped (Die)\n*Exit Code:* {exit_code}"
send_telegram_msg(msg)
if exit_code == "137":
send_telegram_msg(f"⚠️ Warning: {container_name} was OOM Killed (Out of RAM)!")
elif status == "oom":
send_telegram_msg(f"🔥 *CRITICAL*: {container_name} has run out of memory!")
if __name__ == "__main__":
monitor_events()3. Deploying as a Sidecar Container
To let the script run itself and monitor other containers, package it into Docker. Don’t forget to mount the socket inside the docker-compose.yml file, especially if you are optimizing Docker Compose for your production environment:
services:
docker-monitor:
image: my-docker-monitor:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- TELEGRAM_TOKEN=${TELEGRAM_TOKEN}
- CHAT_ID=${CHAT_ID}
restart: alwaysSecurity Warning: Mounting docker.sock grants the script full control over the Docker Engine. You must never make this image public on Docker Hub if it contains sensitive information. You can scan your Docker host for CIS compliance to ensure your setup remains secure.
Upgrade: Smart Automation
Instead of just sending messages, you can add automated handling logic. For example: if a container restarts more than 5 times in 10 minutes, pause it. This helps avoid CPU throttling for the entire server.
You can also use the Events API as an Audit Log. It helps you know exactly who deleted which container and when. In practice, this solution helped me reduce incident response time by 80%. Instead of waiting for a customer call, I receive a message and handle it right from my phone.
In summary, if you manage Docker and haven’t used the Events API, you’re missing out on a top-tier tool. It’s lightweight, free, and flexible for any automation needs.
