A 2 AM Nightmare and the Cost of Manual Rollbacks
The Slack screen was glowing red with notifications from Sentry. The Grafana dashboard showed CPU charts spiking like lightning bolts. This was the nightmare scenario I experienced last week right after hitting the “Merge” button on a new payment feature.
It was 2 AM. A logic error in the promotion processing caused the system to hang. The only way to save it was a rollback. Unfortunately, the company’s CI/CD process took a full 15 minutes to build images, run tests, and push to Kubernetes. 15 minutes of “clinical death” for the system meant over 2,000 canceled orders and a serious hit to our reputation.
The feeling of helplessness, watching the server fail while being able to do nothing but wait for the Jenkins progress bar, was truly haunting.
The Root Cause: When Deployment is Confused with Release
Our core mistake is often letting business logic and the deployment process be tightly coupled.
To enable a feature, you deploy code. To disable it when there’s a bug, you have to roll back the code. This traditional approach carries three major risks:
- Slow response: Waiting for CI/CD is torture when Production is on fire.
- All or nothing: You’re forced to release a feature to 100% of users instead of testing it on a small 5% group.
- Code rot: Blocks like
if (env === 'production')sprout like mushrooms, making the codebase messy.
Are Common ‘Quick Fix’ Solutions Effective?
Many developers immediately think of using Environment Variables. Edit the environment variable and restart the pod. This is faster than a full code deployment but still requires a server restart, causing connection interruptions for users for a few seconds.
Others choose to store configurations in a Database. The system checks the DB for every user request. This is more flexible but inadvertently puts pressure on the database and increases system latency without a caching mechanism.
Unleash – Decoupling Deployment from Release
Learning from that mistake, I decided to set up a proper Feature Flags system. After considering LaunchDarkly (quite expensive at around $75/month for a small team) and Flagsmith, I chose the open-source version of Unleash.
Unleash allows you to push code to production at any time, but new features remain dormant behind a “switch.” When you’re confident, just go to the Unleash UI and flip the switch. Every change takes effect immediately without touching a single line of code.
Setting up Unleash Server with Docker in 5 Minutes
To have full control over data and save costs, I chose to self-host using Docker Compose. This is the optimal solution for engineering teams.
# docker-compose.yml
version: "3.9"
services:
db:
image: postgres:15
environment:
POSTGRES_DB: unleash
POSTGRES_PASSWORD: password
healthcheck:
test: ["CMD", "pg_isready", "-U", "postgres", "-d", "unleash"]
interval: 2s
timeout: 5s
retries: 10
unleash:
image: unleashorg/unleash-server
ports:
- "4242:4242"
environment:
DATABASE_URL: postgres://postgres:password@db/unleash
DATABASE_SSL: "false"
DATABASE_MAX_CONNECTIONS: 20
UNLEASH_SECRET: your-secure-secret
depends_on:
db:
condition: service_healthy
Run the command docker-compose up -d, then access localhost:4242. Log in with username admin and password unleash42 to start managing flags.
Connecting a Node.js App: Near-Zero Latency
Instead of hard-coding, I wrap the logic in the Unleash Client. This library is extremely smart because it fetches flags and caches them in the application’s memory.
Installing the library is simple:
npm install unleash-clientConfiguration in actual code:const { initialize, isEnabled } = require('unleash-client');
const unleash = initialize({
url: 'http://your-unleash-api/api/',
appName: 'payment-service',
customHeaders: { Authorization: 'YOUR_API_TOKEN' },
});
async function processPayment(order) {
// Check 'new-payment-gateway' flag in memory
if (isEnabled('new-payment-gateway')) {
return useNewGateway(order);
}
return useOldGateway(order);
}
Since flags are checked directly from the cache, latency is only about < 1ms. If the Unleash server goes down, the client automatically uses the last cached value or falls back to a default value, ensuring the system is never interrupted.
Real-World Experience to Avoid ‘Technical Debt’
Feature Flags are a powerful tool, but if overused, they can turn code into a mess.
1. The ‘Clean Up After the Party’ Rule
Once a feature is running stably for 100% of users, delete that flag immediately. Don’t let your code get filled with if-else blocks from features that launched ages ago.
2. Gradual Rollout Strategy
Instead of enabling it for everyone, I usually test for 10% of users based on userId. If metrics are stable, I gradually increase to 25%, 50%, and then 100%.
Pro tip: When you need to quickly test complex JSON strategy configurations, I often use toolcraft.app/en/tools/developer/json-formatter to format and verify the data returned from the Unleash API correctly.
3. Naming Flags by Purpose
Forget meaningless names like test-flag-1. Use the format: [action]-[feature]-[version], for example: enable-stripe-v3-migration, so anyone on the team can understand its purpose at a glance.
Conclusion
Since adopting Unleash, I’ve been able to sleep much better. If an issue occurs, I just open my phone, access the dashboard, and flip the switch to OFF. In less than a second, all users are returned to a safe version.
If your project is scaling and deployment frequency is increasing, implement Feature Flags today. Don’t wait until you’re stuck monitoring a server in the middle of the night to regret it.
