When a Container Is “Half-Dead” and Docker Has No Clue
You’ve probably found yourself in this frustrating situation: A client messages you saying the system is down. You open the terminal, run docker ps, and see the container proudly displaying Up 3 days.
The shell is running fine on the outside. But inside, the application is stuck in a deadlock, leaking memory, or exhausting database connections. Every incoming request times out.
This is just like a cashier passing out behind the checkout counter. From afar, someone appears to be sitting in the chair. Customers form a long line, but no one is being served. Docker’s default mechanism only checks the primary process (PID 1). As long as this process hasn’t exited with an error code, Docker assumes everything is running smoothly.
When I first started using Docker Compose, I fell right into this trap. I added restart: always to the configuration file and went to bed with peace of mind. At 2:00 AM sharp, the Node.js app exhausted its 2GB of RAM and froze solid. Because the process never actually crashed, Docker flatly refused to restart it. By morning, my phone was blowing up with urgent messages from management. Back then, the quick fix was simply SSHing in and running a single command: docker restart.
We need a more proactive approach. The system must continuously verify whether the application is genuinely responding to requests. That’s where the combination of Docker Healthcheck and Docker Autoheal comes in to save the day.
The Lifesaving Duo: Healthcheck and Docker Autoheal
To make a container self-healing, you need two components: a diagnostic check and an execution agent.
1. Docker Healthcheck
Healthcheck is a directive defined in a Dockerfile or docker-compose.yml. Periodically, at an interval you specify (such as every 10 seconds), Docker runs a check command inside the container.
- Healthy: The command returns exit code
0(HTTP 200). The app is functioning properly. - Unhealthy: The command fails (exit code
1) past the allowed retry threshold. Docker flags the container as unhealthy.
The fatal flaw: Out of the box, Docker only stops at setting a warning status. Even when seeing an unhealthy container, Docker Compose ignores it and won’t restart it for you.
2. Docker Autoheal
Docker Autoheal (created by Will Farrell) is an ultra-lightweight monitoring container that consumes less than 20MB of RAM. It connects directly to the Docker Socket on the host.
Autoheal periodically monitors container health. The moment it detects an unhealthy container, Autoheal immediately sends a restart command to the Docker daemon. The entire process happens automatically within seconds—no late-night firefighting required.
Configuring Docker Autoheal with Docker Compose
Here is a complete setup for running an Nginx web app monitored by Autoheal.
Step 1: Create the docker-compose.yml File
Create a new directory and open docker-compose.yml:
services:
# 1. Web application to monitor
my-web-app:
image: nginx:alpine
container_name: my-web-app
ports:
- "8080:80"
restart: always
labels:
# Tell Autoheal to monitor this container
- "autoheal=true"
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost/ || exit 1"]
interval: 10s # Check every 10 seconds
timeout: 3s # Wait up to 3 seconds for a response
retries: 3 # Mark as unhealthy after 3 consecutive failures
start_period: 5s # Allow app 5 seconds to warm up before checks
# 2. Container to automatically restart failed services
autoheal:
image: willfarrell/autoheal:latest
container_name: autoheal
restart: always
environment:
- AUTOHEAL_INTERVAL=5 # Poll health status every 5 seconds
- AUTOHEAL_START_PERIOD=10 # Wait 10 seconds before starting checks
- AUTOHEAL_DEFAULT_STOP_TIMEOUT=10 # Grace period for container shutdown
volumes:
# Mount the Docker socket so Autoheal can restart containers
- /var/run/docker.sock:/var/run/docker.sock
Key Parameters Explained
autoheal=true: The mandatory label for Autoheal to detect targets. This prevents Autoheal from inadvertently restarting background batch workers while jobs are in progress./var/run/docker.sock: The communication socket for interacting with the Docker daemon. Security notice: Any container granted access to this socket effectively has root-equivalent privileges on the host.curl -f http://localhost/: The-f(fail silently) flag makes curl return an error code upon receiving HTTP 4xx or 5xx responses, triggering theexit 1condition.
Step 2: Start the Stack
Start both services with the following command:
docker compose up -d
Wait about 15 seconds, then check the running containers:
docker ps
Check the STATUS column for my-web-app; you should see Up ... (healthy). The healthcheck endpoint is working as expected.
Step 3: Test Auto-Recovery in Action
To simulate a web failure (returning 404), temporarily rename the index.html file inside the container:
docker exec -it my-web-app mv /usr/share/nginx/html/index.html /usr/share/nginx/html/index.html.bak
Open another terminal tab and follow Autoheal’s logs:
docker logs -f autoheal
After 3 failed curl attempts (around 30 seconds), Docker marks the container as unhealthy. Autoheal catches the event and intervenes immediately:
Container my-web-app (9a1f2b3c4d5e) found to be 'unhealthy'
Restarting container my-web-app (9a1f2b3c4d5e)...
Container my-web-app (9a1f2b3c4d5e) restarted
The restart command is triggered entirely automatically. Once you’ve verified the restart, restore the original filename to return the app to a healthy state:
docker exec -it my-web-app mv /usr/share/nginx/html/index.html.bak /usr/share/nginx/html/index.html
During real-world incidents such as connection pool exhaustion or heap memory overflow, this automatic restart clears stuck resources and brings the application back online immediately.
Conclusion
You don’t need a heavy Kubernetes cluster just to achieve self-healing capabilities. For small-to-medium systems running on a single VPS, pairing Docker Healthcheck with Docker Autoheal provides a lightweight and remarkably effective solution.
Keep in mind that Autoheal is merely first aid. If logs reveal that Autoheal is restarting containers every hour, you’ll need to debug your code and resolve the root cause once and for all.
