Distributed Locks with Redis and Redlock: Handling Resource Contention in Microservices

Four Strategies for Handling Distributed Resource Contention

Migrating from a monolith to microservices solves scalability challenges, but it introduces major headaches around race conditions. Back in the monolith days, a simple synchronized keyword in Java or a mutex in Go was enough to lock in-memory resources. Today, with 50 Kubernetes pods distributed across 10 physical servers, local memory synchronization is completely off the table.

Imagine a flash sale scenario: 100 iPhone 16 Pro Max units available, but 5,000 requests hit the system in a single second. Without proper resource locking, inventory will immediately go negative. To prevent this disaster, we have four primary options:

• Database Locking (RDBMS): Use SELECT ... FOR UPDATE (pessimistic) or verify a version column (optimistic).
• Single Redis Instance: Leverage Redis’s single-threaded nature and atomic command: SET resource_name my_random_value NX PX 30000.
• Redlock Algorithm (Multi-master): Coordinate lock acquisition across N independent Redis nodes (typically 5) using quorum-based majority consensus.
• Consensus Engines (ZooKeeper / etcd): Rely on heartbeats, leases, and consensus algorithms like Paxos or Raft to guarantee strong consistency (CP in the CAP theorem).

Weighing the Pros and Cons of Each Approach

Each tool is designed for specific throughput limits and risk tolerances. Don’t use a sledgehammer to crack a nut:

1. Database Locking

• Pros: Familiar, leverages existing infrastructure, and guarantees strict ACID compliance.
• Cons: Creates a severe bottleneck at the database layer. Holding transactions open for too long drains the connection pool (often limited to 100–200 connections), triggering cascading failures across the entire system.

2. Single Redis Instance (or Master-Replica)

• Pros: Blazing-fast performance running entirely in-memory with sub-millisecond latency. Extremely simple setup.
• Cons: Redis replication is asynchronous. If the master acquires a lock and crashes before replicating it to the replica, the newly promoted replica won’t know the lock exists. As a result, two clients end up holding the same lock simultaneously.

3. The Redlock Algorithm

• Pros: Completely eliminates the Single Point of Failure (SPOF). A simple majority (e.g., 3 out of 5 nodes) is sufficient to acquire a valid lock, even if 1–2 nodes crash unexpectedly.
• Cons: Requires managing multiple independent instances. Latency increases slightly due to multiple network round trips, and it is sensitive to clock drift.

4. ZooKeeper / etcd

• Pros: Highly reliable thanks to heartbeats and ephemeral nodes. If a client disconnects, the lock is automatically released.
• Cons: Complex operational overhead and resource-heavy. Throughput hovers around a few thousand ops/sec, far behind Redis’s 50,000–100,000 ops/sec.

Which Solution Should You Choose?

After many nerve-wracking on-call shifts, here is a practical rule of thumb:

• Use Single-Node Redis (with Lua Scripts): Ideal when you need massive throughput (tens of thousands of req/sec). Even if an edge-case race condition occurs (<0.1% probability), the impact is negligible and easily reconciled by end-of-day batch jobs.
• Use Redlock or etcd: Essential for payment flows, wallet balance deductions, or scheduled cron jobs that must strictly execute exactly once across the entire cluster.

Quick tip when generating mock data to stress-test your Redis cluster: if you need to quickly convert CSV user or order lists into JSON, check out toolcraft.app/en/tools/data/csv-to-json. The tool runs directly in your browser, so you don’t have to worry about leaking sensitive customer data.

Hands-on: Implementing Distributed Locks

Step 1: Standardizing on Single-Node Redis with Lua Scripts

Developers new to distributed systems often call SETNX followed by EXPIRE. This is dangerous. If the process gets killed between the two commands, the key persists indefinitely, causing a hard deadlock. Always use the SET command with atomic flags:

import uuid
import time
import redis

client = redis.Redis(host='localhost', port=6379, db=0, decode_responses=True)

def acquire_lock(lock_name, acquire_timeout=10, lock_timeout=10):
    """
    lock_timeout: Maximum lock hold duration before auto-expiration (seconds)
    acquire_timeout: Maximum wait time to acquire the lock (seconds)
    """
    identifier = str(uuid.uuid4())  # Prevent accidental release of another process's lock
    lock_key = f"lock:{lock_name}"
    end_time = time.time() + acquire_timeout
    
    while time.time() < end_time:
        # nx=True: set only if not exists; ex=lock_timeout: auto-expire after X seconds
        if client.set(lock_key, identifier, nx=True, ex=lock_timeout):
            return identifier
        time.sleep(0.05)  # Avoid tight spinning to prevent CPU spikes
    return False

When releasing the lock, never call DEL directly. If the task exceeds its time limit (longer than 10 seconds), the original lock will have expired, and another client might have already acquired a new one. A naive DEL would accidentally delete someone else’s lock. Always verify ownership using an atomic Lua script:

RELEASE_LOCK_LUA = """
if redis.call('get', KEYS[1]) == ARGV[1] then
    return redis.call('del', KEYS[1])
else
    return 0
end
"""

def release_lock(lock_name, identifier):
    lock_key = f"lock:{lock_name}"
    result = client.eval(RELEASE_LOCK_LUA, 1, lock_key, identifier)
    return result == 1

Step 2: Upgrading to Multi-Node Redlock

For higher fault tolerance, deploy 5 independent Redis nodes (no master-replica replication, no cluster sharding). The Redlock algorithm operates in 4 steps:

1. Record the current timestamp (T1).
2. Sequentially request the lock from all 5 nodes using a small timeout (5–50ms). This prevents getting blocked by unresponsive nodes.
3. Calculate the elapsed time: elapsed_time = now() - T1. The lock is successfully acquired if at least 3 out of 5 nodes grant it AND the remaining validity time Validity Time = TTL - elapsed_time - clock_drift is greater than 0.
4. If the lock fails to achieve a quorum (fewer than 3 nodes) or expires, immediately send an unlock command to all 5 nodes.

Install the Python library:

pip install redlock-py

Practical implementation:

from redlock import Redlock
import time

# Connect to 5 independent Redis nodes across 5 separate servers
dlm = Redlock([
    {"host": "10.0.0.1", "port": 6379, "socket_timeout": 0.1},
    {"host": "10.0.0.2", "port": 6379, "socket_timeout": 0.1},
    {"host": "10.0.0.3", "port": 6379, "socket_timeout": 0.1},
    {"host": "10.0.0.4", "port": 6379, "socket_timeout": 0.1},
    {"host": "10.0.0.5", "port": 6379, "socket_timeout": 0.1},
], retry_count=3, retry_delay=0.2)

resource_key = "order:payment:20261103"
# Hold lock for 10,000ms (10 seconds)
my_lock = dlm.lock(resource_key, 10000)

if my_lock:
    try:
        print("Lock acquired. Deducting wallet balance...")
        time.sleep(2)  # Process critical payment logic
    finally:
        dlm.unlock(my_lock)
        print("Lock safely released.")
else:
    print("Resource is busy, please try again later!")

3 Critical Production Lessons

• Watchdog Mechanism (Automatic TTL Renewal): A Java GC stop-the-world pause or a slow query can cause a task to exceed 10 seconds, releasing the lock before the operation completes. Implement a background thread (similar to Redisson’s Watchdog) to periodically extend the TTL as long as the process is alive.
• Beware of Clock Drift: Redlock relies on physical wall-clock time to calculate TTL. A clock drift of even 500ms can cause the algorithm to make incorrect decisions. You must enable NTP synchronization across all servers.
• Fencing Tokens: For strict financial transactions, distributed locks alone are not enough. The lock server must issue a monotonically increasing token (e.g., 1, 2, 3…). When writing to persistent storage, the database rejects any transaction bearing a token smaller than or equal to a previously committed token.