The Real Problem: Your docker-compose.yml File Keeps Growing
About six months ago, I was maintaining an e-commerce project with around 8 microservices. The docker-compose.yml file looked reasonable at the time — until the team added Prometheus, Grafana, and Jaeger for observability, then Mailhog for email testing, then pgAdmin to inspect the database…
The result? The docker-compose file ballooned to 300 lines. Every time we deployed to staging, I had to manually comment out a bunch of unnecessary services. Once I forgot to uncomment them — Grafana went straight to production without anyone noticing. Server RAM jumped 40% just because of monitoring tools running where they didn’t need to be.
Then there was the day I discovered a memory leak in a container: it took exactly two days to debug. Part of the reason was that the local and staging environments weren’t consistent — local was running Jaeger tracing, staging wasn’t. The trace logs didn’t match, and I kept chasing the wrong lead.
That’s when I dug into Docker Compose Profiles — a feature that’s been available since Compose v1.28 but that most people overlook.
Root Cause: Stuffing Everything Into One File Is a Design Mistake
We try to use a single docker-compose.yml file for four completely different scenarios:
- Local development: needs hot-reload, debug ports, a mail catcher, DB admin UI
- CI/CD testing: just the app and database, run tests, then tear it down
- Staging: close to production, with monitoring added
- Production: only real services, nothing extra
A lot of people solve this with multiple files: docker-compose.override.yml, docker-compose.prod.yml, and so on. That works. But as the number of services grows, things get messy — you have to remember the merge order, and overlapping configs become easy to get wrong.
Approaches I’ve Tried
Approach 1: Multiple Compose Files With Manual Merging
# Run with multiple files
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
# Or use an override file
docker compose -f docker-compose.yml -f docker-compose.override.yml up -dIt works, but it’s a pain. Every developer has to remember a long command. CI/CD scripts get hardcoded. Adding a new environment means adding a new file and a new script — it multiplies indefinitely.
Approach 2: Environment Variables to Toggle Services
Some people try tricks with environment variables and extends, but Compose doesn’t natively support conditionally enabling or disabling services. The result is hacky code that new team members can’t make sense of.
The Cleanest Approach: Docker Compose Profiles
Profiles are an official feature built exactly for this problem. Each service is assigned one or more profiles. When you run Compose, you declare which profiles are active — only services belonging to those profiles (or services with no profile at all) will start.
Using Docker Compose Profiles in Practice
Basic Structure
# docker-compose.yml
version: '3.9'
services:
# Core services — no profile = always runs
app:
image: myapp:latest
ports:
- "3000:3000"
environment:
- NODE_ENV=production
postgres:
image: postgres:15
volumes:
- pgdata:/var/lib/postgresql/data
redis:
image: redis:7-alpine
# --- Only runs when the "debug" profile is active ---
mailhog:
image: mailhog/mailhog
profiles: ["debug"]
ports:
- "8025:8025"
pgadmin:
image: dpage/pgadmin4
profiles: ["debug"]
ports:
- "5050:80"
environment:
- [email protected]
- PGADMIN_DEFAULT_PASSWORD=admin
# --- Only runs when the "monitoring" profile is active ---
prometheus:
image: prom/prometheus
profiles: ["monitoring"]
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
grafana:
image: grafana/grafana
profiles: ["monitoring"]
ports:
- "3001:3000"
# --- Only used for CI testing ---
test-runner:
image: myapp:test
profiles: ["test"]
command: npm test
depends_on:
- postgres
- redis
volumes:
pgdata:Running for Each Environment
# Production: just app + postgres + redis
docker compose up -d
# Local development: add mailhog + pgadmin
docker compose --profile debug up -d
# Staging with monitoring:
docker compose --profile monitoring up -d
# Need both debug and monitoring at the same time:
docker compose --profile debug --profile monitoring up -d
# CI/CD test run:
docker compose --profile test run --rm test-runnerBesides the --profile flag, you can also set profiles via an environment variable — more convenient when used with a .env file:
# Use the COMPOSE_PROFILES environment variable
export COMPOSE_PROFILES=debug,monitoring
docker compose up -d
# Or set it in your .env file
echo "COMPOSE_PROFILES=debug" >> .envA Service Can Belong to Multiple Profiles
jaeger:
image: jaegertracing/all-in-one
# Runs when either debug OR monitoring is active
profiles: ["debug", "monitoring"]
ports:
- "16686:16686"
- "4317:4317"Inspecting Which Services Belong to Which Profile
# List all services and their profiles
docker compose config --services
# View the fully rendered config
docker compose config
# List services that will run when the "debug" profile is active
docker compose --profile debug config --servicesThe Actual Setup I Use for the E-Commerce Project
version: '3.9'
services:
api:
build: ./api
environment:
- DATABASE_URL=postgresql://user:pass@postgres:5432/ecom
- REDIS_URL=redis://redis:6379
depends_on:
- postgres
- redis
postgres:
image: postgres:15
volumes:
- pgdata:/var/lib/postgresql/data
environment:
- POSTGRES_DB=ecom
- POSTGRES_USER=user
- POSTGRES_PASSWORD=pass
redis:
image: redis:7-alpine
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
- ./ssl:/etc/nginx/ssl
# === DEBUG TOOLS ===
pgadmin:
image: dpage/pgadmin4
profiles: ["debug"]
ports:
- "5050:80"
mailhog:
image: mailhog/mailhog
profiles: ["debug"]
ports:
- "8025:8025"
# Override api to enable debug mode
api-dev:
build:
context: ./api
target: development
profiles: ["debug"]
volumes:
- ./api:/app
- /app/node_modules
environment:
- NODE_ENV=development
- DEBUG=*
ports:
- "9229:9229" # Node.js debug port
# === MONITORING ===
prometheus:
image: prom/prometheus
profiles: ["monitoring"]
volumes:
- ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
grafana:
image: grafana/grafana
profiles: ["monitoring"]
ports:
- "3001:3000"
volumes:
- grafana_data:/var/lib/grafana
cadvisor:
image: gcr.io/cadvisor/cadvisor
profiles: ["monitoring"]
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
volumes:
pgdata:
grafana_data:Integrating Profiles Into Your Real Workflow
.env Files Per Environment
# .env.local
COMPOSE_PROFILES=debug
NODE_ENV=development
# .env.staging
COMPOSE_PROFILES=monitoring
NODE_ENV=staging
# .env.production
COMPOSE_PROFILES=
NODE_ENV=production# Deploy by environment
docker compose --env-file .env.local up -d
docker compose --env-file .env.staging up -d
docker compose --env-file .env.production up -dA Simple Makefile to Wrap the Commands
up-dev:
docker compose --profile debug up -d
up-staging:
docker compose --profile monitoring up -d
up-prod:
docker compose up -d
test:
docker compose --profile test run --rm test-runner
down:
docker compose --profile debug --profile monitoring --profile test downOne easy-to-miss gotcha: When you run docker compose down without specifying a profile, Compose only stops services that have no profile. To stop everything, you need to list all profiles explicitly — or use the --profiles "*" flag available from Compose v2.20+:
# Stop all services, including those with profiles
docker compose --profiles "*" downResults After Six Months of Real-World Use
After migrating to Profiles, the docker-compose.yml file became noticeably cleaner — no more scattered comment blocks everywhere. Developers on the team no longer accidentally spin up monitoring tools in production, because without setting COMPOSE_PROFILES, those services simply don’t exist as far as Compose is concerned.
Remember that memory leak I mentioned at the start? If we’d been using Profiles back then, Jaeger would have run consistently across both local and staging under the same debug profile. The trace logs would have matched, and I wouldn’t have wasted two days chasing the wrong lead.
Profiles don’t solve everything — secrets management and per-environment config still need other tools. But for a team of three or more people running multiple environments, this is the clearest organizational approach I’ve ever used with Docker Compose. Once you try it, going back to the old way is hard to justify.
