Context: When a Naive Webhook Meets a Bad Actor
Back when I was working on an e-commerce project, our team had a webhook that received payment notifications from Stripe. The code was fairly simple: receive POST request → update order → send confirmation email. Then one day, someone started flooding that endpoint with fake requests. Within 15 minutes, the system had processed over 300 fake “successful orders” — along with 300 confirmation emails blasted out to real customers.
The problem wasn’t in the business logic. The problem was that the webhook had no authentication mechanism. Anyone who knew the URL could POST fake data to it.
Looking back, it came down to two issues:
- Forged requests: How do you tell whether a request actually came from Stripe or from an attacker?
- Data loss on retry: Your server goes down for 30 seconds — Stripe retries, you process it twice, the customer gets charged twice.
HMAC-SHA256 solves the first problem. Idempotency plus smart retry logic solves the second. This article focuses on exactly those two things — nothing more.
Setup
Environment: Node.js 18+, Express. No extra HMAC library needed — Node.js ships with the built-in crypto module.
mkdir webhook-secure && cd webhook-secure
npm init -y
npm install express dotenvCreate a .env file — and remember to add it to .gitignore:
WEBHOOK_SECRET=your_super_secret_key_here
PORT=3000WEBHOOK_SECRET is the shared secret between you and the service sending the webhook. With Stripe, you get this key from Dashboard → Webhooks → Signing secret. Guard it like a password.
Detailed Configuration
1. HMAC Verification: Checking the Request’s Digital Signature
How it works: when Stripe sends a webhook, they compute HMAC-SHA256(request_body, secret) and attach it to a header. Your server does the same with the received body — if the two signatures match, the request is legitimate.
The most common mistake: you must parse the raw body (as a Buffer), not already-parsed JSON. If Express parses JSON first, the body string changes whitespace → signatures don’t match → constant 401s even with the right key.
// webhook.js
require('dotenv').config();
const express = require('express');
const crypto = require('crypto');
const app = express();
// Raw body required for HMAC — don't use express.json() here
app.use('/webhook', express.raw({ type: 'application/json' }));
function verifyHmacSignature(payload, signature, secret) {
const expected = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
const sigBuf = Buffer.from(signature, 'hex');
const expBuf = Buffer.from(expected, 'hex');
// timingSafeEqual prevents timing attacks — don't use === here
if (sigBuf.length !== expBuf.length) return false;
return crypto.timingSafeEqual(sigBuf, expBuf);
}
app.post('/webhook', (req, res) => {
const signature = req.headers['x-webhook-signature'];
if (!signature) {
return res.status(401).json({ error: 'Missing signature' });
}
const isValid = verifyHmacSignature(
req.body,
signature,
process.env.WEBHOOK_SECRET
);
if (!isValid) {
console.warn(`[WARN] Invalid signature from ${req.ip}`);
return res.status(401).json({ error: 'Invalid signature' });
}
const event = JSON.parse(req.body.toString());
// Respond 200 immediately — don't let Stripe time out and retry
res.status(200).json({ received: true });
// Handle async processing after responding
processEventIdempotent(event).catch(err =>
console.error(`[ERROR] ${event.id}: ${err.message}`)
);
});Why not use signature === expected? Regular string comparison stops at the first mismatched character. An attacker can send thousands of requests with signatures that differ by one byte at a time, measuring response times to incrementally guess the value — this is a timing attack. timingSafeEqual always runs for a constant amount of time regardless of the comparison result, so it doesn’t leak information this way.
2. Idempotency Key: Process Exactly Once, No Matter How Many Times You Receive It
The webhook sender will retry if it doesn’t receive a 200 within a few seconds. Stripe retries on an escalating schedule over 3 days — starting at 5 seconds and going up to several hours between attempts. In a worst case, you might receive the same event more than a dozen times. Without handling this: double-charging customers, sending emails twice.
// In production, use Redis. Using Set here for simplicity.
const processedEvents = new Set();
async function processEventIdempotent(event) {
const eventId = event.id; // Stripe, GitHub both provide unique event IDs
if (processedEvents.has(eventId)) {
console.log(`[INFO] Duplicate event ${eventId}, skipping`);
return;
}
// Mark as processed before handling — avoids race conditions with concurrent requests
processedEvents.add(eventId);
try {
await processEvent(event);
console.log(`[INFO] Event ${eventId} processed successfully`);
} catch (err) {
// On error, allow retry by removing from Set
processedEvents.delete(eventId);
throw err;
}
}
async function processEvent(event) {
switch (event.type) {
case 'payment.success':
await handlePaymentSuccess(event.data);
break;
default:
console.log(`[INFO] Unhandled event: ${event.type}`);
}
}3. Retry with Exponential Backoff: Handling Downstream Failures
After verification, webhooks typically need to call other APIs — a CRM to update orders, a notification service, an inventory system. If any of those are temporarily down, without smart retries you lose data.
Retry logic has three classic pitfalls: retrying indefinitely when a service is completely down, missing jitter which causes thousands of clients to retry simultaneously (thundering herd), and retrying 4xx errors — which will always fail no matter how many attempts. The code below handles all three.
async function retryWithBackoff(fn, maxRetries = 3, baseDelay = 1000) {
let lastError;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (err) {
lastError = err;
// Don't retry client errors (4xx) — they will always fail
if (err.statusCode >= 400 && err.statusCode < 500) throw err;
if (attempt === maxRetries) break;
// Exponential backoff + jitter: prevents multiple clients from retrying simultaneously
const delay = baseDelay * Math.pow(2, attempt - 1) + Math.random() * 500;
console.log(`[WARN] Attempt ${attempt} failed. Retry in ${Math.round(delay)}ms`);
await new Promise(resolve => setTimeout(resolve, delay));
}
}
throw new Error(`All ${maxRetries} attempts failed: ${lastError.message}`);
}
// Used in handlePaymentSuccess:
async function handlePaymentSuccess(data) {
await retryWithBackoff(async () => {
const res = await fetch('https://crm.example.com/orders', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ orderId: data.orderId })
});
if (!res.ok) {
const err = new Error(`CRM error: ${res.status}`);
err.statusCode = res.status;
throw err;
}
});
}Testing & Monitoring
Testing with curl
Start the server first:
node webhook.js
# Webhook server running on port 3000Compute the HMAC and send a valid request:
PAYLOAD='{"type":"payment.success","id":"evt_123","data":{"orderId":"ORD-456"}}'
SECRET="your_super_secret_key_here"
SIGNATURE=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $2}')
curl -X POST http://localhost:3000/webhook \
-H "Content-Type: application/json" \
-H "x-webhook-signature: $SIGNATURE" \
-d "$PAYLOAD"
# {"received":true}Test with an invalid signature — should return 401:
curl -X POST http://localhost:3000/webhook \
-H "Content-Type: application/json" \
-H "x-webhook-signature: fake_signature_here" \
-d '{"type":"test"}'
# {"error":"Invalid signature"}Test idempotency — send the same event ID twice, the second should be skipped:
# First request
curl -X POST http://localhost:3000/webhook \
-H "Content-Type: application/json" \
-H "x-webhook-signature: $SIGNATURE" \
-d "$PAYLOAD"
# Second request — server log will show "Duplicate event evt_123, skipping"
curl -X POST http://localhost:3000/webhook \
-H "Content-Type: application/json" \
-H "x-webhook-signature: $SIGNATURE" \
-d "$PAYLOAD"Metrics to Monitor in Production
Add logging middleware for easier debugging:
app.use('/webhook', (req, res, next) => {
const start = Date.now();
res.on('finish', () => {
console.log(JSON.stringify({
ts: new Date().toISOString(),
status: res.statusCode,
duration_ms: Date.now() - start,
ip: req.ip
}));
});
next();
});
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Webhook server running on port ${PORT}`));Four numbers to check every day:
- Spike in 401 rate: your endpoint might be getting probed
- Response time > 3 seconds consistently: move processing to a queue
- Rising duplicate event count: upstream is retrying heavily, something is wrong on your end
- Retry exhausted: downstream service is unstable, alert immediately
This setup is production-ready for most use cases. When volume grows to thousands of events per second, push events into a queue (Bull/BullMQ + Redis) instead of processing inline. The HMAC and idempotency logic doesn’t change — just swap processEvent() with queue.add(event) and you’re done.
