I use Python as my automation tool for most daily tasks, from deploy scripts to monitoring alerts. As the project grew, the test suite jumped from 50 tests to 300+, and every git push meant staring at the CI/CD spinner for nearly 10 minutes just waiting for the tests to finish. Not because the code had bugs — but because the tests were genuinely slow.
The Real Problem: A 300-Test Suite Eating 8 Minutes on Every Push
The specific project: a Python toolset made up of multiple modules — file processing, third-party API calls, database queries. The test suite had around 320 tests, with 60% being integration tests checking complex logic with mock HTTP and file I/O.
Results when running the full suite:
$ pytest --tb=no -q
320 passed in 487.23s (8 minutes 7 seconds)8 minutes. Multiply that by 10–15 pushes a day and that’s over an hour just waiting for tests. It’s even worse on CI/CD because CI machines typically have fewer cores than dev machines.
Root Cause Analysis: Pytest Runs Sequentially by Design
Pytest runs tests one at a time by default — each test must finish before the next one starts. This is an intentional design decision to ensure consistency and ease of debugging, but it doesn’t take advantage of multi-core CPUs.
Looking at the resource monitor while tests run on an 8-core machine:
- Total CPU: 12–15% (1 core busy, 7 cores idle)
- Disk I/O: spikes per test for file reads/writes
- Network: occasional mock HTTP response calls
The problem is clear: tests aren’t CPU-heavy, they’re mostly I/O wait time — waiting for mock responses, file writes, DB query returns. During that wait, the other 7 CPU cores sit completely idle. This is a textbook case of wasted resources.
Ways to Fix This
Option 1: Optimize Individual Tests
The first step to take regardless of whether you use xdist. Common tricks:
import pytest
# Mark slow tests to skip when needed
@pytest.mark.slow
def test_heavy_integration():
result = call_real_api() # takes 2-3 seconds
assert result["status"] == "ok"# Local dev: skip slow tests
pytest -m "not slow"
# CI full: run all tests
pytestAfter optimizing, we got down from 8 minutes to ~6 minutes. Still slow, not good enough.
Option 2: Run Only Tests Related to Your Changes
# Run only tests in the recently modified module
pytest tests/test_api_client.py
# Run by keyword name
pytest -k "upload or download"
# Run previously failed tests first
pytest --lf # last failed
pytest --lf --nf # last failed, then new filesUseful for local dev but CI/CD still needs to run the full suite. Doesn’t solve the root problem.
Option 3: Run in Parallel with pytest-xdist
pytest-xdist spawns multiple worker processes, each receiving a subset of the test suite and running in parallel. This is the right fix — instead of waiting sequentially, all CPU cores work together.
The Best Approach: Pytest-xdist from Setup to Production Use
Installation
pip install pytest-xdist
# Or add to requirements-dev.txt
pytest-xdist>=3.0Basic Usage
# Auto-detect CPU core count — this is the command I use most
pytest -n auto
# Specify a fixed number of workers
pytest -n 4
# Combine with verbose output
pytest -n auto -v --tb=shortResults right after enabling it:
320 passed in 127.41s (2 minutes 7 seconds)
# From 8 minutes → 2 minutes, ~4x fasterTest Distribution Strategies with –dist
This is what took me the longest to figure out. pytest-xdist has several distribution strategies:
# load (default): idle workers pick up the next test
pytest -n 4 --dist=load
# loadscope: groups by class or module — good when using class-scoped fixtures
pytest -n 4 --dist=loadscope
# loadfile: all tests in the same file run on the same worker
pytest -n 4 --dist=loadfileI use --dist=loadfile for projects with many complex class fixtures, because it ensures tests in the same file aren’t split across different workers — avoiding quite a few race conditions.
Fixtures and xdist: The Most Important Thing to Watch Out For
This is the most common source of bugs when first using xdist. A scope="session" fixture runs independently on each worker, not shared across the entire session.
# Each worker needs its own port to avoid conflicts
@pytest.fixture(scope="session")
def server_port(worker_id):
# worker_id: "gw0", "gw1", "gw2"... or "master" when not using xdist
base_port = 8000
if worker_id == "master":
return base_port
worker_num = int(worker_id.replace("gw", ""))
return base_port + worker_num # gw0=8000, gw1=8001, gw2=8002...For database tests, I use tmp_path to give each test its own database:
@pytest.fixture
def db(tmp_path):
db_path = tmp_path / "test.db"
conn = sqlite3.connect(str(db_path))
yield conn
conn.close()
# tmp_path is automatically cleaned up after the testCombining with pytest-cov
pip install pytest-covThe .coveragerc file:
[run]
concurrency = multiprocessing
parallel = true
[report]
omit =
*/tests/*
*/venv/*# Run tests with coverage
pytest -n auto --cov=src --cov-report=html
# Combine data from multiple workers and export the report
coverage combine
coverage report -mPractical Tips
Tests must be completely stateless — this is the prerequisite:
# WRONG: global state is shared between workers
counter = 0
def test_increment():
global counter
counter += 1
assert counter == 1 # Fails when running in parallel!
# CORRECT: each test manages its own state
def test_increment():
counter = 0
counter += 1
assert counter == 1Avoid hardcoding ports or fixed file paths:
import socket
def get_free_port() -> int:
with socket.socket() as s:
s.bind(('', 0))
return s.getsockname()[1]
@pytest.fixture
def app_port():
return get_free_port() # each worker gets a random port, no conflictsUse tmp_path instead of creating fixed files:
def test_file_processing(tmp_path):
input_file = tmp_path / "input.csv"
input_file.write_text("col1,col2\n1,2\n3,4")
result = process_csv(input_file)
assert len(result) == 2 # 2 data rowsProfiling to find the slowest tests:
# Show the 20 slowest tests
pytest --durations=20
# Install pytest-randomly to detect order-dependent tests
pip install pytest-randomly
pytest -n auto # tests will be randomly shuffledReal-World Results and Pre-xdist Checklist
After applying pytest -n auto --dist=loadfile, my test suite dropped from 8 minutes → about 2 minutes on an 8-core machine. The CI pipeline went from 30 minutes down to ~12 minutes (the rest is Docker build and deploy).
Before enabling xdist, quickly check these points:
- Each test does not use shared mutable state (global variables, class attributes)
- No hardcoded ports or fixed file paths in tests
scope="session"fixtures have no side effects that affect other tests- Test databases use transaction rollback or isolated in-memory DB / tmp_path
- No tests depend on the execution order of other tests
If any tests break when running in parallel, debug by re-running with -n 1 to confirm they pass sequentially — if -n 1 passes but -n auto fails, it’s definitely a shared state or port/file conflict.
