A Midnight Bandwidth Bottleneck Nightmare: Real-World Experience with a Payment System
At precisely 11:15 PM on a Friday, my phone started buzzing relentlessly with PagerDuty alerts. Our API gateway and payment processing cluster saw latency spike from 45ms all the way to 2.8 seconds. Nginx 504 Gateway Timeout rates hit 18%.
Opening Grafana, the network traffic graph shot straight up in solid red. Network interface bandwidth on the switch was maxed out at 980 Mbps—dangerously close to the 1 Gbps port limit. I rushed to SSH into each server, running iftop and nethogs. But this approach was far too fragmented. In a microservices cluster of nearly 40 VMs, you cannot manually inspect individual processes while an incident unfolds second by second.
The system was severely choked. Who was transmitting this data? What protocol was consuming the bandwidth? Was it outbound internet traffic or bouncing around internally? At that moment, we were flying completely blind.
Why Do Prometheus and Zabbix Fall Short During Bandwidth Chokepoints?
Following the post-mortem, our team dissected the incident and identified a critical gap in our monitoring data:
- Standard metrics (Prometheus node_exporter, Zabbix): They only track raw volumes, such as total bytes sent/received or packets per second. They have no insight into what application-layer (Layer 7) data those packets actually contain.
- Service logs: Nginx access logs or iptables logs only record events once a TCP session closes or a packet is dropped. They are completely oblivious to in-flight flows that are actively lingering and saturating bandwidth.
- Lack of DPI (Deep Packet Inspection): An unexpected rsync backup, a database dump accidentally pushed to S3, or a DNS Amplification attack all look identical to standard traffic unless you inspect the payload headers.
Three Approaches to Network Flow Analysis
To address this challenge, I considered three common options:
- Capturing raw packets with tcpdump and analyzing in Wireshark: Running
tcpdump -i eth0 -w dump.pcapinspects traffic down to the bit level. However, the capture file ballooned to 10 GB in under 3 minutes, exhaustively consuming disk I/O, and real-time analysis during high loads was out of the question. - Shipping NetFlow / sFlow to an ELK stack: The router exports flows to Logstash, which stores them in Elasticsearch. While robust, this setup is notoriously memory-hungry. An Elasticsearch cluster dedicated to this requires at least 16 GB of RAM, involves an overly complex pipeline configuration, and still introduces a 1–2 minute reporting lag.
- Deploying ntopng with nDPI: ntopng captures packets and provides visualization dashboards simultaneously. Powered by its built-in nDPI library, it dissects Layer 7 protocols directly in memory, leverages Redis for ephemeral flow caching, and serves real-time metrics with very modest resource overhead (roughly 500 MB to 1 GB of RAM).
Production Deployment of ntopng on Ubuntu / Debian
Here are the step-by-step instructions for deploying ntopng directly on a dedicated network monitoring server (connected via a switch SPAN/mirror port or sitting directly on a Linux gateway).
Step 1: Install the Repository and ntopng Packages
Use the official repository from ntop rather than the default Ubuntu packages to ensure you have the latest nDPI signatures:
# 1. Install prerequisites
sudo apt update && sudo apt install -y software-properties-common wget lsb-release gnupg
# 2. Download and install the official ntop repository package
wget https://packages.ntop.org/apt/$(lsb_release -rs)/all/apt-ntop.deb
sudo dpkg -i apt-ntop.deb
sudo apt update
# 3. Install ntopng and Redis (used for state caching)
sudo apt install -y redis-server ntopngStep 2: Configure the Target Network Interface
Open the configuration file at /etc/ntopng/ntopng.conf:
sudo nano /etc/ntopng/ntopng.confClear the sample content and apply optimized parameters for production:
# Web UI listening port
-w=3000
# Network interface to capture packets on (verify with 'ip a')
-i=eth0
# Define local subnet ranges so ntopng can differentiate Inbound / Outbound / Local traffic
-m=192.168.1.0/24,10.10.0.0/16
# Directory for storing RRD metrics and analytical data
-d=/var/lib/ntopng
# Disable automatic reverse DNS lookups to prevent thread contention during high traffic
-n=1The -n=1 flag is critical. Under high congestion with tens of thousands of concurrent connections, ntopng’s attempt to resolve DNS for every single IP will freeze the packet capture process.
Step 3: Start the Services and Log In
Start Redis first, then enable and launch ntopng:
sudo systemctl enable redis-server ntopng
sudo systemctl restart redis-server
sudo systemctl restart ntopng
# Verify that the service is running properly
sudo systemctl status ntopngNavigate to http://<IP-Server>:3000. The default credentials are admin / admin. The web interface will require setting a new password upon first login.
Step 4: Overcoming Alert Fatigue — Cutting Through Notification Spam
Our first week running ntopng in production was an alerting nightmare. The system defaulted with every rule enabled: from out-of-order packets and minor jitter to unusual DNS queries. As a result, our Slack bot blasted over 2,300 notifications daily. By day three, the team muted the channel entirely, instantly rendering the monitoring system useless.
To permanently resolve this issue, I fine-tuned three key areas:
- Segment Host Pools: Group internal infrastructure (such as database replication clusters or Ceph storage) into dedicated pools. Under Settings > Alert Entities, suppress high-bandwidth alerts between these nodes, as syncing hundreds of gigabytes between them is standard behavior.
- Raise Flow Flood Thresholds for Gateways: Reverse proxies (Nginx/HAProxy) naturally handle tens of thousands of concurrent connections. Navigate to that host’s threshold settings and increase the Max Number of Flows threshold to 8x the default to prevent ntopng from mistaking legitimate traffic for a SYN flood attack.
- Trigger Only Actionable Alerts: Retain notifications strictly for Blacklisted IP Contacts, TCP Connection Refused bursts (a telltale sign of internal port scanning), and DNS Tunneling. If an alert does not demand immediate engineer intervention, turn it off.
Operational Takeaways
Deploying ntopng finally resolved that midnight bottleneck mystery. The culprit turned out to be a buggy database backup cronjob: instead of compressing files locally and routing them through an internal secondary network, the script streamed an uncompressed 40 GB raw dump straight through the public API gateway.
Today, whenever bandwidth alerts trigger, the team simply checks the Flows > Top Talkers tab. In under 20 seconds, we pinpoint the exact IP hogging bandwidth, the destination port, and the application type—putting an end to hunting in the dark over SSH.
