A Guide to Polars in Python: Processing Big Data and Eliminating OOM Errors to Replace Pandas

Python tutorial - IT technology blog
Python tutorial - IT technology blog

The Incident: 2 AM and the Out-of-Memory (OOM) Death Sentence

At 2:15 AM, my phone buzzed incessantly with PagerDuty alerts. The pod running a transaction aggregation batch job on our Kubernetes cluster had just turned bright red: CrashLoopBackOff.

Rushing to open the terminal and inspect the logs with kubectl describe pod, the all-too-familiar message appeared:

Out of memory: Kill process 28415 (python3) score 942 or sacrifice child
Killed process 28415 (python3) total-vm:18452140kB, anon-rss:15984120kB

The culprit was easy to find: a Python cron job using Pandas to read a 3.2 GB CSV file (~22 million log lines). Even with 16 GB of RAM allocated to the node, it still crashed.

The root cause lies in Pandas’ eager evaluation mechanism. It pulls the entire dataset into memory at once. Combined with Python object overhead, actual RAM usage often balloons to 4 to 5 times the file’s raw size on disk. A single subsequent merge or column creation is enough for the Linux kernel’s OOM Killer to instantly terminate the process.

Throwing more RAM at worker nodes is an expensive band-aid. Instead, I decided to rewrite the pipeline using Polars—a DataFrame library written in Rust on top of Apache Arrow. The killer advantage: Polars leverages multi-core CPUs (unencumbered by Python’s GIL) and employs lazy evaluation to optimize query execution plans before running.

Installing Polars for Production Environments

Polars ships with pre-built binary wheels, so you don’t need a Rust compiler installed in your containers or staging servers.

Install the core package via pip:

pip install --upgrade polars

If your pipeline needs to read/write Parquet or pull data directly from PostgreSQL/ClickHouse, install these additional packages:

pip install pyarrow connectorx fsspec

Note for older VPS instances: the standard Polars build utilizes AVX2 instructions. If deploying on older CPUs lacking AVX2 support, use the compatibility build instead:

pip install polars-lts-cpu

Shifting Mindsets: From Eager to Lazy Execution

Many developers migrating to Polars instinctively call pl.read_csv(). However, this still loads the entire dataset into RAM, behaving just like Pandas. To eliminate OOM once and for all, you must use pl.scan_csv() or pl.scan_parquet() to construct a LazyFrame.

Here is the actual refactored code used to filter error logs and aggregate revenue by user:

import polars as pl

# 1. Initialize LazyFrame - Not a single byte has been read into RAM yet
lazy_df = pl.scan_csv(
    "/var/log/app/transactions_large.csv",
    has_header=True,
    separator=",",
    low_memory=True
)

# 2. Build Query Plan
query = (
    lazy_df
    .filter(pl.col("status") == "FAILED")
    .filter(pl.col("amount") > 0)
    .with_columns([
        pl.col("timestamp").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S"),
        pl.col("error_code").fill_null("UNKNOWN")
    ])
    # Extract txn_id using Regex
    .with_columns(
        pl.col("raw_message").str.extract(r"TXN_([A-Z0-9]+)", 1).alias("txn_id")
    )
    .group_by(["user_id", "error_code"])
    .agg([
        pl.len().alias("fail_count"),
        pl.col("amount").sum().alias("total_failed_amount")
    ])
    .sort("fail_count", descending=True)
)

# 3. Trigger streaming computation: Process in small batches
result_df = query.collect(streaming=True)

# Write results to Parquet format
result_df.write_parquet("/var/log/app/summary_failed.parquet")
print("Job completed successfully without hitting peak RAM limits!")

Quick regex tip: the TXN_([A-Z0-9]+) pattern extracts the transaction ID from raw logs. When debugging urgently in the middle of the night, instead of spending time writing test scripts, I quickly pasted a sample log into the regex tester at toolcraft.app/en/tools/developer/regex-tester to verify capture groups directly in the browser before plugging it into the code.

How Does Lazy Execution Save RAM?

  • Predicate Pushdown: Polars recognizes the status == 'FAILED' filter and applies it directly while reading the file from disk. Successful rows are skipped right at the I/O layer without ever entering memory.
  • Projection Pushdown: Does your file contain 35 columns while the query only needs 5? Polars parses only those 5 required columns and skips the remaining 30 entirely.
  • Streaming Engine: When calling collect(streaming=True), Polars chunks the dataset into manageable batches sized for L3 cache and RAM. As a result, a 50 GB file can easily run on a machine with just 8 GB of RAM.

Benchmark Results: Polars vs. Pandas on a Production Server

Before closing the incident ticket, I ran a benchmark comparing both solutions against the same 3.2 GB log file.

1. Inspecting the Query Plan Before Execution

To verify that Polars has pushed the filtering operations down to disk I/O, print the plan with:

print(query.explain())

If you see SELECTION: None at the top level and PROJECT 5/35 COLUMNS along with PREDICATE at the scan level, your query is properly optimized.

2. Measuring Real-World Resource Usage with psutil

import time
import psutil
import os

def get_mem_mb():
    return psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)

start_time = time.time()
print(f"Initial RAM: {get_mem_mb():.1f} MB")

df = query.collect(streaming=True)

print(f"Peak RAM: {get_mem_mb():.1f} MB")
print(f"Execution time: {time.time() - start_time:.2f}s")

Real-World Results on an 8 vCPU, 16 GB RAM VM

  • Pandas 2.2: Peaked at 15.2 GB RAM with an 84-second runtime. Severe OOM risk if other services run concurrently.
  • Polars 0.20 (Streaming): Peak RAM capped at 820 MB, slashing execution time to 11.2 seconds. All 8 CPU cores were consistently utilized at 90–95%.

Following that incident, our team migrated every file-processing pipeline handling 500 MB or more to Polars. The outcome: our Kubernetes cluster runs smoothly, pod memory limits dropped from 16 GB down to 2 GB, saving us several hundred dollars in cloud costs every month.

Share: