A Guide to DiskCache in Python: Safe Multi-Process Disk Caching Without Redis - ITFROMZERO - Share tobe shared! Boost Python performance on low-RAM servers with DiskCache: safe, high-speed multi-process disk caching without installing or managing Redis.

Table of Contents

Real-World Dilemma: The RAM Bottleneck on a 1GB VPS

A few years ago, I took on a batch job to process financial reports on a cheap VPS with just 1GB of RAM. The script’s role was to scrape exchange rates from four partner APIs, reconcile the data, and compute statistical metrics. Each cycle took about 8 to 10 seconds. Most of that latency stemmed from network round-trips and repetitive data aggregations.

The immediate fix back then: caching. I slapped @functools.lru_cache across the processing functions. Initial results looked very promising. Response times plummeted from 8 seconds to under 20ms for data already held in RAM. However, as the data volume grew, the system ran into three fatal issues:

Data wiped on restart: Every time a patch was deployed or the process restarted, in-memory cache was completely wiped. The system had to shoulder heavy recomputations from scratch.
Killed by the OOM Killer: The dataset expanded from tens of megabytes to 400MB. The VPS ran completely out of memory. The Linux OOM Killer promptly terminated the Python process without even writing a log warning.
Worker fragmentation: Scaling to a multi-process architecture with Gunicorn and Celery (4 workers) meant each worker maintained its own isolated chunk of RAM. Just after Worker 1 finished fetching data, Worker 2 would redundantly make the same API call. CPU cycles were wasted by 3 to 4 times.

Why In-Memory Caching Fails in Production Architectures

The issue isn’t bad code. It stems from the architectural limits of main memory:

Process isolation: Python process memory is isolated by default. Unless you wire up complex multiprocessing.shared_memory structures, processes cannot read each other’s cache.
Lack of persistence: RAM lives and dies with the process. Periodic batch jobs running once every two hours must reload everything from scratch if the cache only resides in RAM.
RAM is far costlier than SSDs: Modern NVMe drives read sequentially at several GB/s and cost dozens of times less than RAM. Stuffing gigabytes of transient data into RAM on low-spec servers is a risky bet.

Three Common Approaches and Their Trade-offs

Before arriving at an optimal solution, most backend engineers cycle through these three approaches:

1. Manually dumping to JSON or Pickle files

This pattern is very familiar: hash the parameters into a filename, then call pickle.dump() to store it under /tmp.

import pickle
import os

def get_cached_report(key):
    path = f"/tmp/cache_{key}.pkl"
    if os.path.exists(path):
        with open(path, "rb") as f:
            return pickle.load(f)
    # Recompute on cache miss...

The pitfall: Race conditions. Two workers writing to the same file in the exact same millisecond will instantly corrupt the pkl file. You also have to roll your own TTL expiration logic and manually clean up when the disk hits 100% capacity.

2. Deploying Redis or Memcached

Redis is the industry gold standard for caching. High speed, great multi-process support, and robust built-in TTL.

The trade-off: Operational overhead. For internal scripts, lightweight scrapers, or cron jobs, installing a Redis daemon, opening network ports, managing firewall security, and maintaining TCP connections is unnecessary friction. The Redis daemon alone consumes 40-70MB of static RAM out of the box.

3. Building a custom cache wrapper on SQLite

SQLite provides rock-solid ACID transactions. However, designing custom schemas, handling serialization/deserialization for complex objects, and implementing LRU eviction logic requires significant maintenance effort.

The Optimal Solution: DiskCache

DiskCache is a pure-Python library built to solve this exact problem cleanly. Under the hood, DiskCache pairs SQLite in Write-Ahead Logging (WAL) mode for metadata management with direct file writes on disk for larger objects. On modern NVMe drives, DiskCache clocks in at 25,000 to 40,000 reads per second. This can even outperform Redis running over localhost because it completely bypasses TCP socket protocol overhead.

When extracting data from logs before caching, you will often need to normalize strings with regex. If you need to quickly test complex regular expressions without opening a separate test script, check out the regex tester at toolcraft.app/en/tools/developer/regex-tester to validate patterns directly in your browser.

Step 1: Installation

pip install diskcache

Step 2: Basic Cache Usage with Key-Value and TTL

The Cache object operates just like a dictionary, but everything is safely synchronized to disk:

from diskcache import Cache
import time

# Initialize cache in the target directory
cache = Cache('/tmp/my_disk_cache')

# Store key with time-to-live (expire = 60 seconds)
cache.set('user_101', {'name': 'Nguyen Van A', 'role': 'admin'}, expire=60)

# Read data back
user = cache.get('user_101')
print("Cached data:", user)

# Check key existence
if 'user_101' in cache:
    print("Key is still valid")

# Check after expiration
time.sleep(61)
print("After 60s:", cache.get('user_101'))  # Returns None

Step 3: Optimizing Heavy Functions with @memoize

Instead of manual get/set calls, simply attach @cache.memoize. The library automatically hashes function arguments into cache keys:

from diskcache import Cache
import time

cache = Cache('./api_cache_dir')

@cache.memoize(expire=3600)  # Cache results for 1 hour
def fetch_partner_rates(date_str: str):
    print(f"[API CALL] Fetching exchange rates for {date_str}...")
    time.sleep(2.5)  # Simulate network latency
    return {"USD": 25450, "EUR": 27300, "date": date_str}

# Call 1: Real execution, takes 2.5s
t0 = time.time()
print(fetch_partner_rates('2026-03-30'))
print(f"Run 1 took: {time.time() - t0:.3f}s")

# Call 2: Read directly from disk, takes under 2ms
t0 = time.time()
print(fetch_partner_rates('2026-03-30'))
print(f"Run 2 took: {time.time() - t0:.3f}s")

Step 4: Ensuring Multi-Process Safety (Process-Safe)

Thanks to SQLite WAL mode and automatic OS-level file locking, multiple child processes can read and write concurrently without race conditions:

from multiprocessing import Pool
from diskcache import Cache
import os

cache = Cache('./shared_cache')

def worker_task(item_id):
    # Independent workers safely access the shared cache directory
    val = cache.get(item_id)
    if val is None:
        val = f"Computed_item_{item_id}_by_pid_{os.getpid()}"
        cache.set(item_id, val, expire=300)
    return val

if __name__ == '__main__':
    # 4 concurrent child processes
    with Pool(processes=4) as pool:
        results = pool.map(worker_task, [10, 20, 30, 10, 20, 30])
        print("Results:", results)

Step 5: Enforcing Storage Quotas with size_limit (Automatic Eviction)

To prevent your disk from ever filling up, specify a cap with the size_limit parameter (in bytes). Once exceeded, DiskCache uses a Least Recently Used (LRU) algorithm to evict the oldest entries:

from diskcache import Cache

# Cap cache at 500MB (500 * 1024 * 1024 bytes)
cache = Cache('./bounded_cache', size_limit=524288000)

# DiskCache automatically purges old records when reaching 500MB

When to Use DiskCache vs. Redis?

Pick the right tool based on your infrastructure architecture:

Choose DiskCache when: Your app runs on a single node, a low-RAM VPS (1-2GB), CLI tools, desktop apps, or cron jobs needing persistent caching across restarts without maintaining a Redis server.
Choose Redis when: You run a distributed microservices architecture where multiple servers must share a central cache, or when you require advanced primitives like Pub/Sub, Sorted Sets, and Redis Streams.

If your system doesn’t need multi-node scaling yet, DiskCache offers a pragmatic approach: clean setup, rock-solid stability, and maximum RAM savings for your server.