2:00 AM and a Broken Multi-Cloud Connection
The PagerDuty screen lit up at 2:15 AM. A data synchronization worker hosted on a Hetzner VPS was repeatedly throwing connection timed out errors while writing to a PostgreSQL cluster on AWS EC2. Both sides were running on Docker. Previously, the team chose to connect over public IPs and whitelist them in Security Groups to save time.
Disaster struck when Hetzner reassigned its IP ranges following scheduled network maintenance. The AWS whitelist became obsolete immediately, paralyzing the entire order processing pipeline for 45 minutes. Exposing port 5432 directly to the public internet risked getting scanned by Shodan bots within 15 minutes. Deploying an AWS NAT Gateway and Direct Connect would inflate infrastructure costs by at least $32 per month per cluster. That was the moment a private mesh network architecture became an absolute necessity.
Three Approaches to Interconnecting Docker Containers Across Servers
When container clusters are scattered across AWS, DigitalOcean, Hetzner, or on-premises office servers, you typically have three options:
- Approach 1: Expose Public Ports and Whitelist IPs (Makeshift)
Map the container port to a public IP (-p 5432:5432) and use UFW or Security Groups to filter traffic. It takes only two minutes to deploy, but maintenance becomes a nightmare when IPs change frequently. - Approach 2: Install WireGuard Directly on the Host OS (Traditional Infrastructure)
Deploy WireGuard on each host, generatewg0.conffiles, and configure iptables rules to route traffic from the Docker bridge into the VPN. It offers top-notch performance, but it can easily conflict with Docker’s network stack during OS updates. - Approach 3: Run Tailscale as a Sidecar in Docker (Container-Level Mesh)
Each service stack runs its own dedicated Tailscale container. The application container binds directly to Tailscale’s network stack using thenetwork_mode: "service:..."directive.
Detailed Comparison of the Three Solutions
| Criteria | Public Port + Whitelisting | Host OS WireGuard | Tailscale Sidecar Container |
|---|---|---|---|
| Security | Low; vulnerable to accidental firewall misconfigurations | Very High | Very High (Zero Trust) |
| Scalability & Maintenance | A nightmare with more than 5 servers | Time-consuming key management and NAT routing | 100% automated via the Control Plane |
| Host Impact | Exposes ports directly on the host | Deep intervention in kernel and iptables | Completely isolated inside Docker |
| Configuration Complexity | Very low | Moderate to high | Just a single docker-compose.yml file |
Why Is a Tailscale Sidecar Better Than a Host Installation?
During a past upgrade from Ubuntu 20.04 to 22.04, the host’s iptables rules were wiped clean. Forwarding rules between the Docker bridge and wg0 vanished entirely, bringing down all communication across three microservices. The team spent three hours in the middle of the night digging through logs just to troubleshoot iptables.
The sidecar pattern eliminates this problem at the root. Networking is packaged together with the application. When migrating containers to a new VPS or switching between cloud providers, you simply run docker compose up -d, and connectivity is established instantly—completely decoupled from underlying host configurations.
Step-by-Step Guide: Configuring a Tailscale Sidecar with Docker Compose
Let’s assume two services need to communicate securely:
- Server A (AWS EC2): Runs a PostgreSQL database.
- Server B (Hetzner VPS): Runs a worker service that needs to query data from Server A.
Step 1: Generate a Tailscale Auth Key
Navigate to your Tailscale Admin Console, go to Settings > Keys > Generate auth key. Enable the Reusable option and assign an ACL tag (for example, tag:server). Tagging prevents the key from expiring after 90 days and automatically applies predefined ACL permissions.
Step 2: Configure the Database Stack on Server A
Create a docker-compose.yml file on Server A with the following configuration:
services:
tailscale-db:
image: tailscale/tailscale:v1.64.0
container_name: tailscale-db
hostname: db-node
environment:
- TS_AUTHKEY=tskey-auth-kXXXXX-XXXXXXXXXXXX
- TS_STATE_DIR=/var/lib/tailscale
- TS_EXTRA_ARGS=--accept-routes=false
volumes:
- ./tailscale-state:/var/lib/tailscale
- /dev/net/tun:/dev/net/tun
cap_add:
- NET_ADMIN
- NET_RAW
restart: unless-stopped
postgres:
image: postgres:15-alpine
container_name: production-postgres
network_mode: "service:tailscale-db"
environment:
POSTGRES_USER: admin
POSTGRES_PASSWORD: SecretPassword123!
POSTGRES_DB: app_db
volumes:
- pgdata:/var/lib/postgresql/data
restart: unless-stopped
depends_on:
- tailscale-db
volumes:
pgdata:
3 Critical Technical Details:
- No
ports:declared: The PostgreSQL container does not expose any ports to the host, remaining invisible to outside scanners. - Shared Network Namespace: The directive
network_mode: "service:tailscale-db"allows PostgreSQL to listen directly on Tailscale’s private IP. - Persistent State Directory: The
./tailscale-statevolume stores the node identity. Without it, restarting the container will generate a duplicate node entry in your Tailscale Admin Console.
Step 3: Start and Verify the Node
Launch the stack on Server A:
docker compose up -d
docker compose logs -f tailscale-db
Once the log displays Boot: ... Success; node is db-node (100.x.y.z), the database cluster has successfully joined your secure mesh network.
Step 4: Connect from Server B
On Server B (which is also joined to the same Tailscale network), verify network path and latency directly:
# Ping directly via MagicDNS; latency typically ranges from 20-35ms between EU datacenters
tailscale ping db-node
# Test the PostgreSQL port over the WireGuard private network
nc -zv db-node 5432
The console will output Connection to db-node 5432 port [tcp/*] succeeded!. The worker connection string can now simply point to postgres://admin:SecretPassword123!@db-node:5432/app_db.
Production Best Practices
- Enforce Least Privilege: Stick to
cap_add: [NET_ADMIN, NET_RAW]. Avoid running withprivileged: trueto reduce the risk of container breakouts. - Leverage MagicDNS: Use the hostname
db-noderather than hardcoding the100.x.y.zIP address, avoiding config updates if the node gets reassigned an IP. - Monitor DERP Relays: The
tailscale statuscommand shows whether a connection isdirect(P2P WireGuard) or routed through arelay(DERP). If relaying occurs, ensure outbound UDP port 41641 is open for optimal throughput.
