When Bandwidth Mysteriously “Disappears” at Midnight
2 AM, the Zabbix alarm screams. Uplink traffic hits 95%, and latency spikes to hundreds of milliseconds. Half-asleep, I SSH into the server, knowing that using <a href="https://itfromzero.com/en/network-en/using-tcpdump-and-wireshark-to-effectively-analyze-network-traffic.html">tcpdump</a> now is suicide – the pcap file would swell to several GBs in just 10 seconds and freeze the system.
Back then, I managed an office of 50 staff and a small server cluster. During an incident, the most critical question isn’t ‘what is the packet content’, but rather: Who is hogging the bandwidth? Where are they going? And what protocol are they using?
NetFlow and IPFIX are lifesavers in these situations. Instead of inspecting every packet, we analyze data flows. I’ll guide you through setting up the nfdump and nfsen duo to turn dry logs into visual charts, helping you resolve issues in a heartbeat.
Don’t Confuse Packet Capture with Flow
Before diving into the configuration, you need to clearly distinguish between these two concepts to avoid using them for the wrong purpose.
What are NetFlow and IPFIX?
Think of NetFlow like a phone bill: you don’t know the conversation’s content, but you know exactly who called whom, when, and for how long. Technically, a ‘Flow’ is defined by a 5-tuple: Source/Destination IP, Source/Destination Port, and Protocol.
- NetFlow: A protocol developed by Cisco, extremely popular. Version v9 is currently the gold standard for enterprise devices.
- IPFIX: Considered the successor to NetFlow v9 but is an open standard (IETF). If you use Mikrotik, Juniper, or Ubiquiti devices, IPFIX is often the default choice.
The biggest advantage is how lightweight it is. Routers only send metadata to the Collector, consuming far fewer CPU resources than running SPAN/Mirroring ports.
Deploying nfdump – The “Workhorse” Data Collection Tool
To collect flows, I chose nfdump. This is a powerful command-line toolset for receiving, storing, and filtering NetFlow data with high performance.
Installing nfdump
On Ubuntu or Debian, installation takes just a few seconds:
sudo apt update
sudo apt install nfdump -yOnce installed, the system will have the nfcapd daemon. It acts as a “base station,” listening on a port (default 9995) to catch data sent from the router.
Configuring a Listener to Catch Flows
Suppose the router (192.168.1.1) pushes flows to the Linux server (192.168.1.10). I’ll run the following command to check the connection:
# Create storage directory (approx. 500MB per day for 50 users)
sudo mkdir -p /var/netflow/router_office
# Run nfcapd in daemon mode
sudo nfcapd -w -D -l /var/netflow/router_office -p 9995Option explanations:
-w: Write data to file every 5 minutes.-D: Run in the background (daemon).-l: Log storage directory.-p: Listening port.
Don’t forget to open the UDP port on the firewall; otherwise, the router can send data all day and the server won’t receive anything.
sudo ufw allow 9995/udpVisualization with nfsen – Instantly Identify Bandwidth Hogs
Command lines are fast, but if you want to present to your boss or monitor weekly trends, you need a visual interface. While nfsen has a somewhat “retro” interface, it is incredibly robust and accurate.
The installation of nfsen is somewhat manual because it’s Perl-based. You’ll need to prepare an Apache and PHP environment:
sudo apt install apache2 php libapache2-mod-php rrdtool librrd-dev -yAfter configuring the nfsen.conf file, you’ll have a dashboard that allows dragging, dropping, and zooming into specific traffic spikes. Looking at it, I can immediately point out: “IP .25 is uploading 20GB of data to Google Drive using a personal account.”.
Real-world Troubleshooting with nfdump Command Line
Back to the 2 AM shift. When the network is lagging, I don’t wait for nfsen to draw a chart; I use nfdump to query directly. This is the fastest way to find the culprit.
Find the top 10 bandwidth-consuming IPs:
nfdump -R /var/netflow/router_office -s ip/bytes -n 10The results show a clear statistics table:
Top 10 IP Addr sorted by bytes:
Date first seen Duration Proto IP Addr Flows(%) Packets(%) Bytes(%)
2023-10-25 02:05:01.123 300.5 ANY 192.168.1.105 450(1.2) 1.2M(45.6) 1.8G(82.4)
...The Bytes(%) column shows that IP 192.168.1.105 is consuming up to 82.4% of the bandwidth (equivalent to 1.8GB in 5 minutes). Crystal clear!
Check what service that IP is using:
nfdump -R /var/netflow/router_office 'host 192.168.1.105' -s port/bytesIf you see port 443 (HTTPS), they might be watching 4K YouTube or downloading large files. If strange ports from 30000 and up appear, be wary of malware or DDoS.
Final Words from the Tech Room
My hard-earned experience: Don’t wait for the “stable door to be bolted after the horse has bolted.” Deploy NetFlow as soon as you set up the system. It not only saves you during 2 AM shifts but also provides ironclad evidence to convince your boss to approve bandwidth upgrade budgets based on actual data.
The combination of nfdump (deep queries) and nfsen (general monitoring) is a high-value combo that every Linux admin should have in their toolkit. I hope these insights help you manage your network infrastructure more easily.
