The Real Problem: ETL Pipeline Runs, Original Data Disappears
I ran into this exact situation: an ETL pipeline kicked off at 2 AM, transformed sales data, and overwrote the existing Parquet file. By morning, the analytics team was complaining that last month’s report numbers had suddenly changed. Digging through the logs turned up nothing because the original data had been completely overwritten.
This isn’t an isolated incident. With mid-sized pipelines, every Parquet overwrite wipes the entire snapshot history — there’s no way to roll back without tedious manual backups beforehand. Many teams work around this by date-stamping their files: sales_2024_11_01.parquet, sales_2024_11_02.parquet… but this approach wastes storage, complicates queries, and still provides no ACID guarantees.
Why Plain Parquet Isn’t Enough
Parquet is excellent for analytics workloads — columnar storage, efficient compression, fast reads. But Parquet is a pure file format with no concept of transactions or versioning. When you overwrite a Parquet file:
- No change history (audit trail)
- No rollback if a job fails midway
- Schema evolution is painful — adding or removing columns can easily break downstream pipelines
- Concurrent writes from multiple jobs lead to race conditions and data corruption
Parquet isn’t at fault here — it does exactly what it was designed to do. What’s missing is a metadata management and transaction layer sitting on top.
How Apache Iceberg + DuckDB Solve This
Iceberg was built to fill exactly that gap. The underlying format is still Parquet, but Iceberg adds a metadata management layer with four core features:
- Snapshot-based versioning: Each write creates a new snapshot without deleting old ones
- ACID transactions: No more concurrent write corruption
- Schema evolution: Add, remove, or rename columns without rewriting all data
- Time Travel: Query data at any point in the past
DuckDB serves as the query engine: lightweight, fast, and runs right on your laptop. Since version 0.10+, the iceberg extension officially supports reading Iceberg tables directly. The entire stack runs locally — no AWS, no Hive Metastore.
Setting Up the Environment
All you need is Python 3.9+ — no other service dependencies.
# Create virtual environment
python -m venv iceberg-env
source iceberg-env/bin/activate
# Install PyIceberg and DuckDB
pip install "pyiceberg[pyarrow,duckdb]" duckdb pandas pyarrowVerify versions after installation:
import pyiceberg
import duckdb
print(pyiceberg.__version__) # >= 0.7.0
print(duckdb.__version__) # >= 0.10.0Creating Your First Iceberg Catalog and Table
PyIceberg requires a catalog to manage metadata. For a local environment, SqlCatalog stores metadata in SQLite — no additional services to set up.
from pyiceberg.catalog.sql import SqlCatalog
import pyarrow as pa
import os
os.makedirs("warehouse/data", exist_ok=True)
catalog = SqlCatalog(
"local",
**{
"uri": "sqlite:///warehouse/catalog.db",
"warehouse": "file://warehouse",
}
)
catalog.create_namespace("analytics")
schema = pa.schema([
pa.field("order_id", pa.int64()),
pa.field("customer_id", pa.int64()),
pa.field("amount", pa.float64()),
pa.field("status", pa.string()),
pa.field("created_at", pa.timestamp("us")),
])
table = catalog.create_table("analytics.orders", schema=schema)
print("Table created:", table.identifier)Writing Data and Creating Snapshots
import pandas as pd
from datetime import datetime, timedelta
# Batch 1: Yesterday's data
df1 = pd.DataFrame({
"order_id": [1, 2, 3],
"customer_id": [101, 102, 103],
"amount": [150.0, 230.5, 89.99],
"status": ["completed", "pending", "completed"],
"created_at": [datetime.now() - timedelta(days=1)] * 3
})
table.append(pa.Table.from_pandas(df1))
snapshot_1 = table.current_snapshot().snapshot_id
print(f"Snapshot 1 ID: {snapshot_1}")
# Batch 2: Add new orders
df2 = pd.DataFrame({
"order_id": [4, 5],
"customer_id": [104, 105],
"amount": [320.0, 45.0],
"status": ["completed", "refunded"],
"created_at": [datetime.now()] * 2
})
table.append(pa.Table.from_pandas(df2))
snapshot_2 = table.current_snapshot().snapshot_id
print(f"Snapshot 2 ID: {snapshot_2}")At this point, Iceberg holds both snapshots. The original data from snapshot 1 is still intact on disk, neither deleted nor overwritten — this is exactly what plain Parquet cannot do.
Time Travel Queries with DuckDB
Now for the most valuable part. DuckDB reads Iceberg tables directly, including querying by specific snapshot:
import duckdb
conn = duckdb.connect()
conn.execute("INSTALL iceberg; LOAD iceberg;")
# Query current state — all 5 orders
result = conn.execute("""
SELECT * FROM iceberg_scan('warehouse/analytics/orders')
ORDER BY order_id
""").fetchdf()
print("Current state (5 rows):")
print(result)
# Time Travel — query snapshot 1 (first 3 orders only)
result_v1 = conn.execute(f"""
SELECT * FROM iceberg_scan(
'warehouse/analytics/orders',
snapshot_id = {snapshot_1}
)
ORDER BY order_id
""").fetchdf()
print(f"\nSnapshot {snapshot_1} — 3 rows only:")
print(result_v1)If the 2 AM pipeline overwrites something by mistake, querying the previous snapshot immediately retrieves the original data — no backup restoration, no downtime.
Schema Evolution — Adding Columns Without Breaking Existing Pipelines
from pyiceberg.types import FloatType
# Add discount_pct column to schema
with table.update_schema() as update:
update.add_column("discount_pct", FloatType())
# Old snapshots remain readable — new column returns NULL for old data
result = conn.execute("""
SELECT order_id, amount, discount_pct
FROM iceberg_scan('warehouse/analytics/orders')
""").fetchdf()
print(result)
# discount_pct = NaN for the 5 existing orders — completely validWith plain Parquet, adding a column like this typically breaks readers for old snapshots due to schema mismatch. Iceberg handles this automatically — the new column returns NULL for old data, and existing readers aren’t broken.
Tips and Best Practices from Real-World Experience
1. Set a Snapshot Retention Policy from the Start
Snapshots accumulate over time and consume disk space. Expire them periodically:
from datetime import datetime, timedelta
# Keep snapshots from the last 7 days
table.expire_snapshots(
older_than_ms=int((datetime.now() - timedelta(days=7)).timestamp() * 1000)
).commit()
print("Expired old snapshots")2. Partition Tables by Time for Faster Queries
Declare the partition spec when creating the table (you can’t add it later with Iceberg v1):
from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform
# source_id=5 is the index of the created_at column in the schema
spec = PartitionSpec(
PartitionField(
source_id=5,
field_id=1000,
transform=DayTransform(),
name="created_day"
)
)
# Declare when creating the table
table = catalog.create_table(
"analytics.orders_partitioned",
schema=schema,
partition_spec=spec
)DuckDB will only scan the necessary partitions instead of reading all files. For a 100GB table, a query targeting a single day can be 20–50x faster than a full scan.
3. Normalize Source Data Before Ingestion
When ingesting data from multiple sources, I often need to convert CSV to JSON for normalization before loading into Iceberg. For files under 10MB, I use a browser-based converter at toolcraft.app/en/tools/data/csv-to-json — it runs client-side so sensitive data never leaves your machine, which is far more convenient than writing a one-off script for each source.
4. Combine with dbt for Complex Transforms
DuckDB has an official dbt adapter (dbt-duckdb). The entire stack runs fully locally: Iceberg storage + DuckDB compute + dbt transforms = a mini data warehouse at zero cloud cost. Perfect for small teams or proof-of-concept work before scaling to production.
The Best Way to Get Started
If you’re managing data pipelines with Parquet files and have ever lost data or lost track of history, I’d recommend migrating to Iceberg incrementally:
- Start with your least critical table and test Time Travel in a dev environment
- Use
SqlCatalogwith SQLite for local environments — no complex infrastructure needed - Set an
expire_snapshotspolicy from day one, don’t put it off - Once comfortable, migrate to a REST Catalog or AWS Glue for production if needed
Iceberg isn’t the right tool for every scenario — for small tables under a few million rows, the metadata management overhead isn’t worth it. But when data grows to tens of gigabytes, or when an audit trail is mandatory (finance, compliance), this is the best solution I’ve used. Since switching to this stack, I no longer lose sleep debugging pipelines.
