Configuring Docker daemon.json: Prevent Disk Full Errors and Production Server Crashes – ITFROMZERO

Table of Contents

The 2 AM 100% Disk Usage Nightmare

Have you ever stayed up all night because a server suddenly crashed? I’ve been there: the VPS reported 100% disk usage, even though the application was just a simple Node.js web app. After checking with the du -sh /var/lib/docker command, I was shocked to find the log directory taking up over 50GB.

The mistake was keeping Docker’s default configuration. In reality, the Docker Engine is designed for compatibility across many environments, not to run optimally out-of-the-box on production. To take control of this system, you must master the /etc/docker/daemon.json file.

Quick Start: A Standard Configuration for Better Sleep

Below is the configuration file I usually apply to servers running Ubuntu 20.04/22.04. You can copy this into /etc/docker/daemon.json to immediately fix common issues.

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  },
  "live-restore": true,
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 65535,
      "Soft": 65535
    }
  }
}

Don’t forget to check for syntax errors before applying:

sudo dockerd --validate-config

If the result is empty (no errors), restart the service:

sudo systemctl restart docker

Technical Analysis: Why Do These Parameters Matter?

1. Storage Driver: Absolute Priority for overlay2

Older versions of Docker often used aufs or devicemapper. However, overlay2 is now the gold standard thanks to its efficient layer management and superior I/O speed. It helps reduce CPU load and saves Inodes on the hard drive.

Note: If you change the driver on a running server, Docker will not find old images. You will need to re-pull all data from scratch.

2. Logging Driver: Limiting to Survive

By default, Docker records everything a container outputs (stdout/stderr) into JSON files and keeps them forever. For an app with 1,000 requests per second, log files can grow by several GBs per day.

"max-size": "20m": Limits each log file to a maximum of 20MB.
"max-file": "5": Keeps only the 5 most recent files (totaling 100MB per container).

This way, you have data for debugging without ever worrying about running out of disk space.

3. Live Restore: Maintaining Uptime During Maintenance

Normally, when you restart the Docker daemon for system updates, all containers are shut down. The live-restore feature allows containers to keep running even while the daemon is restarting. This is vital for ensuring zero-downtime service.

Advanced Resource Optimization

Moving Storage to a Secondary Drive (data-root)

Most VPS providers partition / quite small by default. If you attach an additional 500GB SSD at /data, point Docker storage there to avoid crashing the OS when data expands.

{
  "data-root": "/data/docker-data"
}

Limiting File Descriptors (ulimits)

An Nginx server or a Database handling large amounts of connections will need to open many files simultaneously. The default limit is often just 1024, which is too low for high-load environments. Setting nofile to 65535 ensures container stability and avoids “Too many open files” errors that cause app crashes.

Practical Operational Experience

After years of system management, I’ve derived three invariable rules:

JSON is unforgiving: Just one extra comma at the end of the file, and Docker will refuse to start. Always use the validate command or the jq tool to check.
Centralized Management: Configure Logging and Ulimits at the Daemon level instead of writing them into each Docker Compose file. This ensures every container on the server follows a common standard.
Fixed DNS Configuration: If a container fails to call an external API, try forcing DNS to Google or Cloudflare directly in daemon.json:
```
"dns": ["8.8.8.8", "1.1.1.1"]
```

Spending 5 minutes optimizing daemon.json from the start will save you hours of troubleshooting later. Check your server today!