Simulating Network Failures with Pumba: Don’t Let Your App Crash Over 1% Packet Loss – ITFROMZERO

Table of Contents

What Happens When the Network Gets a “Cold”?

Dev and DevOps folks are likely familiar with this scenario: the app runs smoothly on local and staging, but as soon as it hits production, customers complain, “Why is the system just spinning?”

I once handled an unforgettable incident. A microservices system was running fine until a switch at the data center failed, causing about 5% packet loss. Service A calling Service B timed out repeatedly. Threads were blocked en masse, triggering a domino effect that brought down the entire system in 10 minutes. That’s when it hit me: The network is never perfect.

In reality, networks always have inherent latency, packet loss, or reordering. If an app isn’t designed to handle these “floods,” it will collapse at the slightest hiccup. For deep packet analysis during these events, Scapy is an excellent tool for network troubleshooting.

Why Do Apps Fail During Network Instability?

The problem usually isn’t business logic; it’s how we manage connections. Here are three common scenarios:

Excessive Default Timeouts: Many libraries set default timeouts to 30s or even infinity. When the network lags, requests hang, workers are fully occupied, and resources are exhausted.
“Naive” Retry Mechanisms: Retrying at high frequency while the network is congested? You’re basically DDoS-ing your own system.
Missing Circuit Breakers: Downstream services are slow, but upstream services keep hammering them with requests. Without a breaker, the failure spreads system-wide.

Instead of praying for stability, we should proactively inject faults during testing (Chaos Engineering). For Docker, Pumba is the most effective tool for the job.

Traditional Network Simulation Methods

Before Pumba, engineers typically used a few manual methods:

The tc (Traffic Control) command: A native Linux tool. It’s powerful but has a cryptic syntax. One wrong parameter could disconnect your entire host machine.
Proxy/Gateway Configuration: Using an intermediate proxy to throttle bandwidth. This is time-consuming to set up and hard to automate in CI/CD pipelines.
Pulling the Network Cable: Too “physical” and impossible to apply to containers or cloud environments.

Pumba – The Secret Weapon for Docker Containers

Pumba is an open-source Chaos Testing tool built specifically for Docker. It allows you to challenge containers by killing, stopping, or directly interfering with network interfaces. While Pumba simulates software-level issues, ethtool is used for checking the status and performance of the actual network interface. The best part? You don’t need to change a single line of code.

Pumba runs as a standalone container. It mounts docker.sock to control other containers. Essentially, Pumba wraps the power of tc into simple, easy-to-understand commands, making it easier than manually configuring traffic mirroring or other complex traffic control tasks.

When I need to quickly calculate subnets to limit IP ranges for containers, I often use toolcraft.app/en/tools/developer/ip-subnet-calculator. Just enter the CIDR to get the network range and host count—very handy for isolating network test environments.

1. Quick Installation

You don’t need to install anything on the host. Run Pumba directly via Docker to check the version:

docker run --rm gaiaadm/pumba pumba --version

2. Simulating Latency

Suppose you have a container named web_app. You want to simulate a 3000ms (3-second) outgoing network delay for 5 minutes:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gaiaadm/pumba pumba netem \
  --duration 5m \
  delay --time 3000 \
  web_app

To make it more realistic, add the --jitter parameter. For example: delay --time 3000 --jitter 500. This causes latency to fluctuate between 2500ms and 3500ms, mimicking a real network environment.

3. Simulating Packet Loss

Packet loss is a nightmare for TCP. It forces retransmissions, drastically increasing latency. To simulate a 20% packet loss:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gaiaadm/pumba pumba netem \
  --duration 3m \
  loss --percent 20 \
  web_app

4. Bandwidth Limiting

If you want to test how your app behaves on a weak network (like the days of 2G modems), use the rate command. This is a practical way to understand Linux bandwidth monitoring under constrained conditions:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gaiaadm/pumba pumba netem \
  --duration 5m \
  rate --rate 100kbit \
  web_app

Real-world Scenario: Testing Self-healing Capabilities

I usually apply Pumba to my testing workflow like this:

Spin up the entire stack (App, Redis, Postgres) using Docker Compose.
Use Pumba to completely cut the connection or create a 10-second delay for the database.
Observe the logs: Does the app report clear errors? Is the connection pool overflowing? Most importantly, when Pumba stops, does the app automatically reconnect or just hang?

For example, a scenario creating an unstable network for api_service: every 10 seconds, it lags by 500ms for a duration of 2 seconds:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gaiaadm/pumba pumba \
  --interval 10s \
  netem --duration 2s delay --time 500 api_service

This test helped me discover many connection pool libraries that keep “dead connections” indefinitely if maxLifetime or idleTimeout aren’t configured correctly.

Conclusion

We can’t expect network infrastructure to be 100% stable. However, you can control how your application reacts to failures. Pumba helps you build a more resilient and durable system, much like configuring multi-WAN failover ensures your internet remains “never-down”.

Don’t wait for a production crash to find the cause. Incorporate Pumba into your integration tests or experiment directly on staging. Happy testing, and may your systems stay stable despite the lag!