A Classic Problem: Why Are Customers Charged Twice?
Imagine you’re running a payment system. A customer clicks the confirmation button, but due to network jitter, the Consumer finishes processing but fails to send the ACK back to RabbitMQ before the connection drops. RabbitMQ detects the disconnection and redelivers that message to another Consumer. The result? The customer is charged twice. This isn’t a code bug; it’s a fundamental characteristic of distributed systems.
RabbitMQ commits to “at-least-once delivery” (guaranteed delivery at least once), but it does not guarantee “exactly-once”. Message duplication is inevitable. To solve this, we need to transform the Consumer into an Idempotent Consumer. Simply put: whether it receives a message once or 100 times, the system state remains unchanged.
Environment Setup
To follow along, you’ll need Node.js and Docker to quickly spin up RabbitMQ and Redis. We use Redis as a message_id store because its read/write speeds are extremely fast, making it ideal for checking duplicates within milliseconds.
# Quickly run RabbitMQ and Redis
docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management
docker run -d --name redis -p 6379:6379 redis:alpineInitialize the project and install the necessary libraries:
mkdir rabbitmq-idempotency && cd rabbitmq-idempotency
npm init -y
npm install amqplib ioredis uuidThe Strategy: Check First, Process Later
The workflow is straightforward: every outgoing message must include a unique messageId. When the Consumer receives it, it checks Redis. If the ID already exists, skip it immediately. If not, proceed with processing and store that ID in Redis.
1. Producer: Sending Messages with Identifiers
Never send a “naked” payload. Wrap your data in an object containing metadata for better management.
const amqp = require('amqplib');
const { v4: uuidv4 } = require('uuid');
async function sendOrder() {
const conn = await amqp.connect('amqp://localhost');
const channel = await conn.createChannel();
const queue = 'order_queue';
const message = {
id: uuidv4(), // Unique identifier for each transaction
data: { orderId: 'ORD-999', amount: 500000 }
};
await channel.assertQueue(queue, { durable: true });
channel.sendToQueue(queue, Buffer.from(JSON.stringify(message)), {
persistent: true
});
console.log(`[x] Sent order: ${message.id}`);
setTimeout(() => conn.close(), 500);
}
sendOrder();2. Consumer: Anti-duplication Mechanism with SETNX
Here, we use the Redis SETNX (Set if Not Exists) command. This is an atomic operation. It ensures that even if 10 instances receive the same message simultaneously, only one instance will gain the right to process it.
const amqp = require('amqplib');
const Redis = require('ioredis');
const redis = new Redis();
async function consume() {
const conn = await amqp.connect('amqp://localhost');
const channel = await conn.createChannel();
const queue = 'order_queue';
await channel.assertQueue(queue, { durable: true });
channel.prefetch(1);
channel.consume(queue, async (msg) => {
if (!msg) return;
const { id, data } = JSON.parse(msg.content.toString());
// Try writing to Redis with a 24-hour TTL to avoid memory bloat
const isNew = await redis.set(`msg:${id}`, 'processing', 'NX', 'EX', 86400);
if (isNew) {
try {
console.log(`[v] Processing message: ${id}`);
await processOrder(data); // Simulate business logic
channel.ack(msg);
} catch (err) {
console.error("Processing error:", err);
await redis.del(`msg:${id}`); // Delete key to allow retry
channel.nack(msg, false, true);
}
} else {
console.warn(`[!] Duplicate detected: ${id}. Skipping...`);
channel.ack(msg); // Still need to ACK to remove from queue
}
});
}
async function processOrder(data) {
return new Promise(res => setTimeout(res, 1000));
}
consume();In practice, I once operated a system processing over 1 million messages per day. The key lesson learned is that TTL (Time To Live) is crucial. Without a TTL, Redis will bloat tremendously after a few months, wasting resources and slowing down lookup speeds.
Monitoring and Metrics
Don’t just write the code and forget about it. You need to track these metrics:
- Redis Hit/Miss Rate: If the duplication rate suddenly spikes (e.g., > 5%), there might be a serious network issue.
- Consumer Lag: Monitor this on the RabbitMQ UI. If Unacknowledged Messages rise, it means your processing logic is slower than the incoming message rate.
- Dead Letter Queue (DLQ): Always maintain a “graveyard” for messages that fail too many times. Don’t let a faulty message clog the entire processing pipeline.
Building distributed systems means learning to embrace uncertainty. Instead of trying to prevent duplication absolutely, design your Consumer to be smart enough to recognize and reject what it has already processed.
