Posted inPython

Python Integration Testing: Ending the “Fake Mock, Real Bug” Nightmare with Testcontainers

Python tutorial - IT technology blog
Python tutorial - IT technology blog

Why use the “real deal” instead of Mocks?

Have you ever experienced a situation where tests pass 100% locally, but as soon as you deploy to staging, SQL errors start flying everywhere? I’ve tasted this bitterness myself from over-relying on Mocks. We often mock databases or Redis to make tests run faster. However, Mocks are often “too well-behaved” and don’t reflect the harsh realities of a production environment.

Take for example: A query using Postgres’s JSONB features or complex Window Functions. A Mock will never tell you that you’ve written the wrong syntax. Only when facing a real Postgres instance will the error reveal itself. In one project I worked on, switching from Mocks to real testing helped reduce Database-related bugs by 40% right from the CI stage.

Testcontainers was created to solve this problem. It allows you to spin up an entire Postgres, Redis, or Kafka container into your test environment with just a few lines of Python code.

Quick Start: Running Tests with a Real Postgres Instance in 5 Minutes

First, you need Docker installed on your machine. Don’t worry about manually running docker pull; Testcontainers handles everything. Install the following libraries:

pip install pytest testcontainers sqlalchemy psycopg2-binary

Here is how you initialize a “proper” database right within your test case:

import pytest
from testcontainers.postgres import PostgresContainer
from sqlalchemy import create_engine

def test_postgres_connection():
    # Initialize Postgres 15 container
    with PostgresContainer("postgres:15") as postgres:
        # Get dynamic connection string (ports are mapped randomly to avoid conflicts)
        engine = create_engine(postgres.get_connection_url())
        
        with engine.connect() as conn:
            result = conn.execute("SELECT 1").fetchone()
            assert result[0] == 1
            print("\nPostgres connection successful!")

When you run the pytest command, Testcontainers automatically pulls the image from Docker Hub. It spins up the container, runs the test, and cleans up after itself. The first run might take 30 seconds to download the image, but subsequent runs will be lightning fast.

Leveling Up with Pytest Fixtures

Using the with block as shown above is convenient for small scripts. However, if you have 50 test files and every file spins up a new container, it will be very slow. To optimize, we use Pytest Fixtures with scope="session".

Structure of the conftest.py file

Consolidate your infrastructure setup into a conftest.py file to be shared across the entire project:

import pytest
from testcontainers.postgres import PostgresContainer

@pytest.fixture(scope="session")
def postgres_db():
    # Container is initialized only once for the entire test session
    postgres = PostgresContainer("postgres:15")
    postgres.start()
    
    yield postgres
    
    postgres.stop() # Clean up after all tests have finished

Now, writing tests becomes effortless. You just need to pass postgres_db as an argument to your function:

def test_create_user(postgres_db):
    db_url = postgres_db.get_connection_url()
    # Implement logic to save user to the real DB here
    assert "postgres" in db_url

Combining Database and Redis (Full Stack Testing)

For modern FastAPI apps, combining Postgres for storage and Redis for caching is common practice. Testcontainers supports mixing multiple container types seamlessly.

from testcontainers.redis import RedisContainer
import redis

@pytest.fixture(scope="session")
def redis_client():
    with RedisContainer("redis:7") as container:
        host = container.get_container_host_ip()
        port = container.get_exposed_port(6379)
        client = redis.Redis(host=host, port=port)
        yield client

def test_cache_logic(redis_client):
    redis_client.set("session_key", "active")
    assert redis_client.get("session_key") == b"active"

When running on GitHub Actions, just ensure the Runner supports Docker. Your test environment will now be identical to Production, completely eliminating those strange bugs that only appear on the server.

Battle-tested Tips: Don’t Let Your Tests Crawl

When the number of test cases grows from 10 to 1000, speed becomes a critical issue. Here are a few tips I’ve gathered from various projects:

  • Prioritize Session Scope: Always share containers across the entire test suite. Restarting Postgres for every test function can increase execution time tenfold.
  • Cleanup Strategy: Since the DB is shared, data from a previous test case might break the next one. Use a fixture to TRUNCATE tables after each test function instead of deleting the container.
  • Configure Wait Strategy: Sometimes a container reports it is running, but the DB inside is still initializing. Use wait_for to ensure the database is actually ready to receive connections.
  • Debug via Logs: If a test fails mysteriously, use container.get_logs(). Sometimes the error lies in the DB hitting RAM limits or incorrect encoding configurations that Mocks would never point out.

Switching to Testcontainers might add a few minutes to your test suite’s startup time. However, the peace of mind knowing your code has run against a real Database is priceless. You’ll no longer have to awkwardly explain: “But it worked on my machine!” every time there’s a production bug.

Share:
Tags: