When 98% Code Coverage Still Leaks Bugs into Production
Over six months ago, my team refactored the checkout module for an e-commerce platform spanning over 50,000 lines of TypeScript. Looking at the Jest coverage report, everyone breathed a sigh of relief: 98% line coverage and 94% branch coverage. We all assumed the codebase was bulletproof.
Reality had other plans. On our very first release, a critical logic bug slipped straight into production: VIP customers were allowed to apply double discount vouchers, bringing a 2-million-VND order down to zero. Ironically, on our CI pipeline, all 420 test cases were glowing green.
Digging into the test files, I uncovered the fatal flaw: Code coverage only tracks whether a line of code is executed. It has no clue whether your expect() or assert() statements actually catch errors when the logic breaks.
Many tests merely called functions for the sake of it, mocked data recklessly, or skipped asserting return values altogether just to inflate coverage metrics for the report. That was when I decided to introduce Mutation Testing and Stryker to expose the true quality of our test suite.
Core Concepts: How Does Mutation Testing Work?
While unit tests verify whether code meets business logic requirements, Mutation Testing does the opposite: It intentionally breaks your code to see if your test suite notices.
The process follows three straightforward steps:
- Generate Mutants: Stryker modifies the TypeScript AST (Abstract Syntax Tree) to introduce intentional small bugs. For example: changing
>to>=,a + btoa - b, removing a function call, or flippingtruetofalse. Each faulty version of the code is called a Mutant. - Run the test suite: Stryker reruns the unit tests relevant to the mutated code.
- Evaluate the results:
- Killed: The test suite FAILS. This is the desired outcome—the test immediately caught the broken code.
- Survived: The logic was mutated, but the tests still PASSED. This proves your test cases missed the bug or have superficial assertions.
- Timeout: The mutant caused an infinite loop, exceeding the execution timeout (typically counted as killed).
The key metric produced is the Mutation Score:
Mutation Score = (Killed Mutants / Total Mutants) * 100%
A project might boast 95% Code Coverage, yet its actual Mutation Score could be as low as 35%–40%, meaning more than half of its business logic lacks meaningful test coverage.
Hands-on: Installing and Configuring Stryker in TypeScript
Here are the step-by-step instructions I implemented on a Vitest codebase (you can also adapt this easily for Jest).
Step 1: Create a Sample Module to Highlight Test Gaps
Let’s say we have a function calculating membership discounts in src/discount.ts:
export interface Customer {
tier: 'SILVER' | 'GOLD' | 'PLATINUM';
yearsActive: number;
}
export function calculateDiscount(orderAmount: number, customer: Customer): number {
if (orderAmount <= 0) {
return 0;
}
let discountRate = 0;
if (customer.tier === 'PLATINUM' || customer.yearsActive >= 5) {
discountRate = 0.2; // 20% discount
} else if (customer.tier === 'GOLD') {
discountRate = 0.1; // 10% discount
}
return orderAmount * discountRate;
}
And here is the test file tests/discount.test.ts hastily written to boost coverage:
import { describe, it, expect } from 'vitest';
import { calculateDiscount } from '../src/discount';
describe('calculateDiscount', () => {
it('covers all branches', () => {
// Test branch orderAmount <= 0
expect(calculateDiscount(-10, { tier: 'SILVER', yearsActive: 1 })).toBe(0);
// Test PLATINUM branch
const platinumResult = calculateDiscount(100, { tier: 'PLATINUM', yearsActive: 1 });
expect(platinumResult).toBeGreaterThan(0);
// Test GOLD branch
const goldResult = calculateDiscount(100, { tier: 'GOLD', yearsActive: 2 });
expect(goldResult).toBe(10);
});
});
Running vitest run --coverage displays a pristine 100% green report. However, the customer.yearsActive >= 5 branch is never tested independently for non-PLATINUM customers. Worse, the assertion for PLATINUM merely checks toBeGreaterThan(0) rather than validating the exact value of 20.
Step 2: Install Stryker Mutator
Install the core package alongside the corresponding test runner:
npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner
If your project uses Jest, switch to @stryker-mutator/jest-runner.
Step 3: Set Up stryker.config.json
Create a stryker.config.json file at your project root:
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"packageManager": "npm",
"reporters": ["html", "clear-text", "progress"],
"testRunner": "vitest",
"coverageAnalysis": "perTest",
"mutate": [
"src/**/*.ts",
"!src/**/*.d.ts",
"!src/**/index.ts"
],
"thresholds": {
"high": 80,
"low": 60,
"break": 60
}
}
Pay special attention to three configuration settings:
coverageAnalysis: "perTest": An essential optimization. Stryker only reruns tests that actually cover the mutated line of code, cutting execution time from 15 minutes down to under 2 minutes.mutate: Target only files containing core business logic, strictly excluding types, interfaces, or barrel export files.thresholds.break: If the Mutation Score drops below 60%, the process exits with code 1 to block merging.
Step 4: Run Stryker and Analyze the Results
Add a script to your package.json:
"scripts": {
"test:mutate": "stryker run"
}
Execute the command:
npm run test:mutate
The terminal immediately pinpoints two surviving mutants:
#1. [Survived] EqualityOperator
src/discount.ts:12:47
- customer.yearsActive >= 5
+ customer.yearsActive > 5
#2. [Survived] BinaryExpression
src/discount.ts:13:20
- discountRate = 0.2;
+ discountRate = 0.0;
Stryker flipped >= 5 to > 5 and replaced the 0.2 discount with 0.0. The entire test suite still PASSED. The code was broken, but the tests failed to catch it.
Step 5: Rewrite Tests to Kill Surviving Mutants
We add test cases covering boundary values to tests/discount.test.ts:
import { describe, it, expect } from 'vitest';
import { calculateDiscount } from '../src/discount';
describe('calculateDiscount', () => {
it('returns 0 when order amount is less than or equal to 0', () => {
expect(calculateDiscount(0, { tier: 'SILVER', yearsActive: 1 })).toBe(0);
expect(calculateDiscount(-100, { tier: 'SILVER', yearsActive: 1 })).toBe(0);
});
it('applies exactly 20% discount for PLATINUM customers', () => {
const result = calculateDiscount(100, { tier: 'PLATINUM', yearsActive: 1 });
expect(result).toBe(20); // Assert exact value instead of toBeGreaterThan
});
it('applies 20% discount for customers active for 5+ years even if not PLATINUM', () => {
// Test boundary value at exactly 5 years
expect(calculateDiscount(100, { tier: 'SILVER', yearsActive: 5 })).toBe(20);
// Customers with 4 years do not qualify
expect(calculateDiscount(100, { tier: 'SILVER', yearsActive: 4 })).toBe(0);
});
it('applies 10% discount for GOLD customers with under 5 years', () => {
expect(calculateDiscount(100, { tier: 'GOLD', yearsActive: 2 })).toBe(10);
});
});
Run npm run test:mutate again:
Mutation score based on treated mutants: 100.00%
All mutants were killed!
Open reports/mutation/mutation.html, and the entire file tree is now green. Every flawed logic variation was successfully caught by the tests.
Key Takeaways After 6 Months in Production
Integrating Stryker into our team’s daily pipeline taught us three valuable lessons:
- Avoid scanning the entire repository: The biggest drawback of Mutation Testing is resource consumption. Running it across 50,000 lines of code could take over 40 minutes and pin CPU usage at 100%—a massive waste of resources.
- Isolate core business logic: Configure Stryker strictly for critical business modules such as
src/modules/pricing,src/modules/payment, orsrc/modules/auth. This targeted approach keeps runs under 90 seconds. - Integrate smartly into CI: Rather than running on every commit, we use the
--since origin/mainflag in GitHub Actions. Stryker only inspects changed files within a pull request, keeping CI overhead under 2 minutes per PR.
High code coverage only proves you wrote a lot of test code. Mutation Testing is the true metric that reveals whether those tests will actually safeguard your system when things go wrong.
