Background & Why You Need a Circuit Breaker in Microservices
Imagine your e-commerce system consists of five microservices communicating with one another. Suddenly, a third-party payment gateway experiences an outage. The normal response time of around 150ms suddenly spikes to 30 seconds before timing out.
Meanwhile, thousands of user requests flood in. Because each request hangs waiting for a response, Node.js rapidly exhausts its socket pool and hits the file descriptor limit (the EMFILE error). Memory usage skyrockets, and incoming requests are rejected en masse. Ultimately, the Order, Cart, and Auth services collapse one after another like dominoes. This disaster is known as a Cascading Failure.
The root problem is that distributed networks are inherently unpredictable. When a downstream service is struggling under heavy load, piling on more traffic only accelerates its demise and takes down the caller service alongside it. Reducing the timeout to 2 seconds merely frees requests slightly faster; it does not stop thousands of doomed requests from continuously draining CPU and network resources.
To solve this decisively, we apply the Circuit Breaker Pattern. Much like a household electrical breaker that trips during a short circuit to protect the power grid, a software circuit breaker halts outbound calls to an unhealthy service. It operates across three distinct states:
- Closed: The normal operating state. All requests are routed straight to their destination. When the failure rate exceeds a defined threshold (e.g., above 50%), the breaker automatically trips to Open.
- Open: The system completely halts outbound calls. Incoming requests fail immediately (fail fast) or trigger a fallback handler in under 1ms, consuming zero socket resources while waiting.
- Half-Open: After a predetermined cooldown period (resetTimeout, typically 10–30 seconds), the breaker allows a few canary requests to pass through. If these requests succeed, the circuit resets to Closed and normal operations resume. Otherwise, if errors persist, the circuit immediately trips back to Open.
In production, configuring circuit breaker parameters should always be driven by actual traffic metrics rather than guesswork. Misconfigured settings can inadvertently trip the breaker on perfectly healthy services.
Installing Opossum in a Node.js Project
Within the Node.js ecosystem, opossum is currently the most mature and widely adopted Circuit Breaker library. It provides native Promise/async-await support, a lightweight event-driven architecture, and seamless integration with Prometheus metrics.
Initialize the project and install opossum along with axios:
mkdir node-circuit-breaker-demo
cd node-circuit-breaker-demo
npm init -y
npm install opossum axios
Configuring the Opossum Circuit Breaker
To ensure the breaker functions as intended, you need to understand each parameter thoroughly. An overly sensitive configuration causes false trips over minor millisecond network jitters. Conversely, a configuration that is too lenient reacts far too late when a target service is already down.
Here is a module wrapping a real-world payment API call:
// paymentClient.js
const axios = require('axios');
const CircuitBreaker = require('opossum');
// Function to perform the actual API call
async function callPaymentGateway(orderData) {
const response = await axios.post('https://api.payment-mock.com/charge', orderData, {
timeout: 2500 // Local Axios timeout for each request
});
return response.data;
}
// Circuit Breaker configuration
const breakerOptions = {
timeout: 3000, // Considered failed if it exceeds 3000ms
errorThresholdPercentage: 50, // Open circuit when >= 50% of requests fail
resetTimeout: 10000, // Wait 10 seconds in Open state before retrying
rollingCountTimeout: 10000, // 10-second rolling window to calculate failure rate
rollingCountBuckets: 10, // Split the window into 10 buckets (1 second/bucket)
volumeThreshold: 5 // Require a minimum of 5 requests in the rolling window before tripping
};
const paymentBreaker = new CircuitBreaker(callPaymentGateway, breakerOptions);
// Fallback: Return fallback data when circuit is Open or request fails
paymentBreaker.fallback((orderData, error) => {
return {
status: 'DEFERRED',
message: 'Payment system is currently busy; order queued for background processing.',
orderId: orderData.orderId,
fallback: true
};
});
module.exports = paymentBreaker;
Key Parameters Explained:
- volumeThreshold: Prevents false tripping during low traffic periods. For example, if a single request times out at 3 AM, the error rate is 100% (1/1). Without a minimum volume threshold, the breaker would trigger unnecessarily.
- rollingCountTimeout: Tracks the failure rate over a real-time rolling window instead of accumulating stats indefinitely since application startup. This ensures errors from 30 minutes ago do not skew decisions made right now.
- fallback: Implements graceful degradation. Instead of dumping a 500 error onto the user, the system returns a polite response or enqueues a job into RabbitMQ/Kafka for deferred retry.
Testing & Monitoring Circuit Breaker States
To verify that the circuit breaker behaves correctly, we can listen to Opossum lifecycle events and simulate a failing target endpoint.
// server.js
const paymentBreaker = require('./paymentClient');
// Listen to events to monitor state
paymentBreaker.on('open', () => {
console.warn('⚠️ [CIRCUIT BREAKER] Circuit is OPEN! Blocking outbound calls, triggering fallback immediately.');
});
paymentBreaker.on('halfOpen', () => {
console.info('🔄 [CIRCUIT BREAKER] Circuit switched to HALF-OPEN. Sending probe requests...');
});
paymentBreaker.on('close', () => {
console.info('✅ [CIRCUIT BREAKER] Circuit is CLOSED. Downstream service has fully recovered.');
});
paymentBreaker.on('fallback', (result) => {
console.log('🛡️ [FALLBACK TRIGGERED] Fallback result returned:', result.orderId);
});
// Simulate 15 consecutive requests against an unresponsive endpoint
async function simulateRequests() {
for (let i = 1; i <= 15; i++) {
try {
console.log(`\n--- Sending Request #${i} ---`);
const result = await paymentBreaker.fire({ orderId: `ORDER_${i}`, amount: 100 });
console.log('Result:', result);
} catch (err) {
console.error('Request failed:', err.message);
}
// 800ms delay between calls
await new Promise(resolve => setTimeout(resolve, 800));
}
}
simulateRequests();
Run the test:
node server.js
When executing the script against a non-existent mock domain, you will observe a clear execution lifecycle:
- From requests 1 to 5, Axios consistently times out after 2.5 seconds.
- By the 5th request, the
volumeThresholdcondition is satisfied, and the failure rate hits 100%. Theopenevent triggers immediately. - From the 6th request onward, Opossum cuts off all outbound network traffic. Fallback responses return near-instantaneously with sub-millisecond latency, completely relieving Node.js socket and memory pressure.
- After exactly 10 seconds (resetTimeout), the circuit transitions to
halfOpento probe the downstream service’s health.
For production deployments, you should export Opossum metrics to Prometheus via prom-client. Paired with Grafana dashboards and Alertmanager notifications routed to Slack or Telegram, your SRE team can detect downstream outages well before end users start reporting issues.
