Docker Compose Include: A “Divide and Conquer” Strategy for Thousand-Line YAML Files in Microservices

The Nightmare of a “Gigantic” docker-compose.yml

If you have ever worked on a real-world microservices project, you are likely familiar with the scene of a long, rambling docker-compose.yml file. I once managed an e-commerce system with 15 microservices and 5 databases. The result was a configuration file that swelled to over 800 lines of code.

The consequences were clear: searching for information was extremely exhausting. Just one small indentation error was enough to crash the entire system. In a team, having 5-7 people editing a thousand-line YAML file frequently caused Git merge “disasters.” One time, I wasted an entire morning just resolving configuration conflicts for a mess of tangled services.

Why Does Your Compose File Keep Getting Bigger?

Actually, it’s not Docker’s fault; it’s how we “stuff” it. Initially, the project only had a Web and DB, so one file was enough. You can dockerize your project easily at the start, but as the scale grows, you’ll need a bunch of other things:

• Queue systems: RabbitMQ, Redis, Kafka.
• Separate business services: Auth, Order, Payment, Shipping.
• Monitoring tools: Prometheus, Grafana.
• Log management: ELK stack or Loki.

Many people have the mindset of “gathering everything in one place for easy running.” This exact thinking turns configuration files into an unmaintainable mess, especially when attempting zero-downtime blue-green deployment.

Efforts to Break It Down Before the “Include” Feature

Before the release of Docker Compose V2 (version 2.20.0 and above), we typically used two methods, but both had limitations.

Method 1: Using Multiple Files with the -f Flag

You split it into docker-compose.db.yml and docker-compose.app.yml. When starting, you have to type a long command like this:

docker compose -f docker-compose.yml -f docker-compose.db.yml -f docker-compose.app.yml up -d

Forget just one -f flag, and you’re in trouble. One container won’t see another, or volume mounts might end up in the wrong place, leading to data loss.

Method 2: Using “extends”

The extends keyword helps reuse configuration between services. However, it only solves part of the problem. You cannot use it to cleanly modularize an entire group of resources, including networks and volumes.

The Optimal Solution: Modularization with the “include” Feature

I moved my entire project stack to Docker Compose V2 and felt extremely relieved. The include feature allows you to separate related service groups into their own files, providing a seamless Docker development experience. Then, you just “nest” them into the main file professionally.

What’s the best thing about “include”?

The biggest difference is the ability to automatically handle relative paths. When including a file from another directory, Docker Compose automatically understands that the build context or env_file paths must be calculated from the location of that included file. You no longer have to manually fix paths every time you move a file.

Real-world Example: Microservices Project Structure

Suppose I organize the project into Infrastructure (DB, Redis) and Application (Web, Worker) as follows:

.
├── docker-compose.yml
├── infrastructure/
│   └── db-stack.yml
└── services/
    └── app-stack.yml

The content of the infrastructure/db-stack.yml file is very concise:

services:
  postgres:
    image: postgres:15-alpine
    volumes:
      - db_data:/var/lib/postgresql/data

volumes:
  db_data:

And here is how the docker-compose.yml file at the project root calls everything:

name: my-ecommerce-platform

include:
  - infrastructure/db-stack.yml
  - services/app-stack.yml

services:
  gateway:
    image: nginx:alpine
    ports:
      - "80:80"
    depends_on:
      - web

Now, with just a single docker compose up -d command, Docker will automatically merge all modules into a complete stack. This approach is highly effective when managing multiple remote servers via SSH using Docker Context.

4 Reasons to Start Using “include” Today

1. Clear Separation of Concerns: The DevOps team manages the infrastructure folder, while the Dev team focuses on services. No one steps on each other’s toes when coding.
2. Lightning-Fast Reuse: You can create a standard DB configuration (Gold Standard) and include it in 10 different projects in 30 seconds.
3. High-Speed Debugging: If there’s an error, you only need to open that specific module file. The main file now acts as an overall “map.”
4. Smart Network Management: The name property in the root file ensures that all containers from different modules are automatically on the same default network.

Hard-earned Practical Experience

Through the implementation process, I’ve gathered a few small notes to help you avoid mistakes:

• Always set a project name using name: in the root file for easier container management with the docker ps command.
• Leverage separate .env files for each module to prevent environment variable overlapping.
• Don’t split things too small. Group them by business logic (e.g., Payment group, Shipping group) instead of separating every single service into its own file.

Switching to include has saved me at least 30% of my time whenever I need to scale a project. If you’re tired of a messy pile of YAML, or worrying about Docker Secrets being exposed, apply it now. This isn’t just a technique; it’s a smarter way to manage our systems.