Posted inNetwork

Mastering Nginx Stream Module: TCP/UDP Load Balancing for Databases, Redis, and MQTT

Network tutorial - IT technology blog
Network tutorial - IT technology blog

When Load Balancing is Not Just for Web

I once had a close call when a management system for 50 employees suddenly hit a bottleneck. At the time, I only viewed Nginx as a pure Web Server for running PHP or Node.js.

Whenever load balancing (LB) for Databases or Redis came up, I would immediately scramble to install HAProxy. However, operating too many different tools made debugging a nightmare when issues arose. Then, I discovered the Nginx Stream Module — an incredibly powerful feature that handles connections at the Transport layer (Layer 4).

If you’re struggling with an overloaded MySQL cluster or need a Single Entry Point for thousands of IoT devices connecting to an MQTT Broker, Nginx Stream is the most cost-effective and efficient solution. Instead of installing new software, leverage your existing Nginx instance to orchestrate traffic for your Database or Redis. This approach significantly reduces infrastructure complexity.

Verifying and Enabling the Nginx Stream Module

Trust me, don’t just rush to copy the configuration and run it. Not every Nginx build comes with the Stream Module pre-installed. Default builds on Ubuntu often separate this module to keep the Nginx core as lightweight as possible. To see if your system is ready, run this command:

nginx -V 2>&1 | grep --color -o with-stream

If the output is empty, you’re missing the --with-stream module. On Ubuntu/Debian environments, adding it is quite simple:

sudo apt update && sudo apt install libnginx-mod-stream -y

Important note: Open your /etc/nginx/nginx.conf file and check if the line include /etc/nginx/modules-enabled/*.conf; exists at the very beginning. Without this line, Nginx won’t even look at the modules you just installed.

Standard Configuration: Don’t Confuse it with HTTP

This is a “classic” mistake that can cost IT pros an entire evening to fix. Standard web configurations live within the http { ... } block. However, TCP/UDP configurations must be independent, sitting at the same level as the http block. The standard structure looks like this:

user www-data;
worker_processes auto;

events {
    worker_connections 1024;
}

# Dedicated area for TCP/UDP
stream {
    include /etc/nginx/conf.d/stream/*.conf;
}

http {
    # Website configuration still goes here
    ...
}

For professional management, I always create a dedicated directory /etc/nginx/conf.d/stream/. Each service, like MySQL or Redis, will have its own separate configuration file within it.

Real-world Applications: MySQL, Redis, and MQTT

1. Load Balancing for MySQL Clusters

Suppose your system has two MySQL Read-Replica nodes. The goal is to point the application to a single Nginx IP on port 3306. Nginx will use the Least Connections algorithm to push traffic to the least busy node.

# File: /etc/nginx/conf.d/stream/mysql.conf
upstream mysql_servers {
    least_conn;
    server 10.0.0.10:3306 max_fails=3 fail_timeout=30s;
    server 10.0.0.11:3306 max_fails=3 fail_timeout=30s;
}

server {
    listen 3306;
    proxy_pass mysql_servers;
    proxy_connect_timeout 5s;
    proxy_timeout 60s; # Crucial for heavy queries
}

Pro tip: Don’t set proxy_timeout too low. For reporting queries that take 40-50 seconds, a 30s timeout will drop the connection midway, causing the application to throw constant errors.

2. Failover for Redis

With Redis used as a cache, speed is the top priority. If you’re not using Sentinel but still want a backup plan, use the backup keyword. The secondary node only kicks in when the primary node goes down.

upstream redis_backend {
    server 10.0.0.20:6379;
    server 10.0.0.21:6379 backup;
}

server {
    listen 6379;
    proxy_pass redis_backend;
}

3. Orchestrating 10,000 MQTT Devices for IoT

MQTT is a TCP-based protocol. When the number of sensor devices surges, a single broker often hits CPU bottlenecks. I use the hash mechanism to ensure a device always maintains a stable connection to a specific broker.

upstream mqtt_cluster {
    hash $remote_addr consistent;
    server 10.0.0.30:1883;
    server 10.0.0.31:1883;
}

server {
    listen 1883;
    proxy_pass mqtt_cluster;
}

Security: Preventing Brute Force and IP Leakage

When using Nginx as a proxy, the backend servers only see the Nginx IP. If your application requires the client’s real IP for auditing, look into the Proxy Protocol. However, remember to check if your current versions of MySQL or Redis support this protocol.

To prevent password-guessing (Brute Force) attacks on the Database port, I always limit concurrent connections. For example, each IP is allowed a maximum of 5 connections to the DB:

stream {
    limit_conn_zone $binary_remote_addr zone=db_limit:10m;

    server {
        listen 3306;
        limit_conn db_limit 5;
        proxy_pass mysql_servers;
    }
}

Testing and Monitoring

After editing, run nginx -t to check the syntax. If everything looks good, reload the service with systemctl reload nginx. To confirm Nginx is listening on the desired ports, I usually use ss:

ss -tlnp | grep -E '3306|6379|1883'

By default, Nginx Stream is quite “quiet” regarding logs. You should define a custom log format to easily track bytes sent/received and session duration:

log_format tcp_stats '$remote_addr [$time_local] ' 
                     '$protocol $status $bytes_sent $bytes_received ' 
                     '$session_time';
access_log /var/log/nginx/stream_access.log tcp_stats;

Mastering Nginx Stream has allowed me to cleanly handle complex infrastructure challenges without consuming extra resources for middleware. I hope these insights help you optimize your systems more smoothly.

Share:
Tags: