Real-World Challenges When Building an AI PoC
Late last quarter, our team was assigned an urgent task: build a multimodal search PoC for a catalog of 200,000 fashion products. Users could upload a quick snapshot or type a brief description to find matching items based on both style and color.
The first instinct for any backend engineer was to spin up a standalone Milvus or Qdrant cluster. However, infrastructure reality hit us on day one:
- Heavy RAM consumption: Most standalone vector databases keep the entire HNSW index in RAM to achieve sub-10ms latency. When we tested ingesting 150,000 512-dimensional CLIP vectors along with metadata, the container consumed over 6GB of RAM. Our 8GB dev server quickly started swapping and crashed.
- Massive DevOps overhead: Every developer on the team had to pull multi-gigabyte Docker images, configure port mappings, and manage health checks. When deploying the demo to a cheap staging VPS, the service repeatedly crashed due to the OOM killer.
- Cumbersome metadata filtering: The dataset contained both vectors and tabular metadata like price, category, and inventory levels. Combining semantic search with hard filters on dedicated vector engines often required cumbersome dual-index configurations.
For a PoC or internal microservice, spinning up a distributed cluster is overkill. An embedded database was what our team truly needed.
Why Are Most Vector Databases So Resource-Heavy?
Looking deeper into the architectural layer, traditional vector database systems usually run into two major bottlenecks:
1. In-Memory Indexing
The HNSW (Hierarchical Navigable Small World) algorithm constructs a graph connecting vector points. Its critical drawback is that this entire graph must reside in main memory. For 768-dimensional vectors (such as text-embedding-3-small or BERT), one million records can consume anywhere from 4GB to 8GB of RAM just for the graph structure alone, excluding the raw data.
2. Client-Server Architecture Latency
The client-server model requires applications to communicate with the database over HTTP or gRPC. This introduces network hops and binary serialization overhead. If you only want to integrate semantic search into an offline Python script or CLI tool, requiring a background daemon is a major roadblock.
Current Approaches to Vector Storage
To eliminate OOM issues while staying on schedule for the PoC delivery, our team evaluated three alternatives:
1. In-Memory Libraries (FAISS, Chroma in-memory)
Meta’s FAISS is blazingly fast and completely free. You can persist the index to a .bin or pickle file.
Limitations: FAISS lacks SQL-style metadata filtering, does not provide ACID transactions, and makes updating individual records difficult.
2. RDBMS Extensions (pgvector)
Adding pgvector to an existing PostgreSQL instance is a familiar choice for many backend teams.
Limitations: HNSW on pgvector consumes just as much RAM as dedicated engines. You still need a standalone Postgres instance, making it impossible to neatly package it as an embedded library inside your application.
3. LanceDB — Embedded Vector Database Built on the Lance Format
LanceDB takes an approach identical to SQLite or DuckDB: running in-process directly within the application. Data is written straight to a local directory or object storage (S3, Cloudflare R2).
Solution: Implementing LanceDB for Multimodal Search
LanceDB solved our exact pain points thanks to three key technical features:
- Disk-based ANN Index: Built on the columnar Lance format (compatible with Apache Arrow). This architecture is optimized for random reads on SSDs, allowing you to query directly from disk without loading all vectors into RAM.
- Zero Infrastructure: No Docker required, no port configurations needed. Just install via
pipand run. - Native Multimodal Support: Store text, vectors, image URLs, and metadata in a single, unified schema.
Step 1: Set Up the Environment
Create a virtual environment and install the required packages:
python3 -m venv venv
source venv/bin/activate
pip install lancedb pyarrow sentence-transformers pillowStep 2: Initialize a Local Database
Simply provide a directory path, and LanceDB will automatically initialize the corresponding file structure:
import lancedb
# Connect to database stored in a local directory
db = lancedb.connect("./data/lancedb_demo")
print("Existing tables:", db.table_names())Step 3: Build the Data Ingestion Pipeline (Text & Image)
Here is the complete pipeline: using a CLIP model to encode text into 512-dimensional vectors, then persisting them directly into LanceDB.
import lancedb
from sentence_transformers import SentenceTransformer
# 1. Load CLIP model supporting both text and images
print("Loading CLIP-ViT-B-32 model...")
model = SentenceTransformer('clip-ViT-B-32')
# 2. Sample product data
products = [
{"id": 1, "name": "Men's running shoes", "category": "shoes", "price": 1200000, "text": "black cushioned sports running shoes marathon"},
{"id": 2, "name": "Waterproof windbreaker", "category": "clothing", "price": 650000, "text": "lightweight waterproof jacket rain motorcycle outdoor"},
{"id": 3, "name": "Shockproof laptop backpack", "category": "accessories", "price": 450000, "text": "15.6 inch laptop backpack gray office waterproof"},
{"id": 4, "name": "Women's white sneakers", "category": "shoes", "price": 890000, "text": "white flat women sports sneakers school work"}
]
# 3. Generate vector embeddings from text
texts = [p["text"] for p in products]
embeddings = model.encode(texts)
for i, product in enumerate(products):
product["vector"] = embeddings[i].tolist()
# 4. Ingest data into LanceDB
db = lancedb.connect("./data/lancedb_demo")
tbl = db.create_table("products", data=products, mode="overwrite")
print(f"Successfully ingested {len(tbl)} records.")Step 4: Hybrid Querying (Vector Search with SQL Filtering)
LanceDB supports filtering data using standard SQL syntax directly during vector distance computation:
user_query = "workout shoes under 1 million"
# Encode query to 512-dimensional vector
query_vector = model.encode(user_query).tolist()
# Vector search with pre-filter: filter by price before comparing vectors
results = (tbl.search(query_vector)
.where("price < 1000000", prefilter=True)
.limit(2)
.to_pandas())
print("\n--- SEARCH RESULTS ---")
for _, row in results.iterrows():
print(f"Product: {row['name']} | Price: {row['price']:,}đ | Distance: {row['_distance']:.4f}")Step 5: Create an IVF-PQ Index at Scale
When your table scales to tens of thousands or millions of rows, build an IVF-PQ (Inverted File with Product Quantization) index to accelerate queries without bloating RAM usage:
# Create IVF-PQ index on vector column
# Note: Requires at least a few thousand vectors for the algorithm to effectively train clusters
tbl.create_index(
metric="cosine",
num_partitions=64, # Number of Voronoi partitions
num_sub_vectors=16 # Quantize original vector into 16 sub-vectors
)Production Lessons Learned
After deploying the PoC to staging, our team took away three important lessons:
- Leverage Serverless Architecture with S3: You can pass an S3 URI like
s3://my-bucket/lancedbdirectly to the connect function. All logic runs in AWS Lambda while data resides in S3, keeping infrastructure costs close to zero when there is no traffic. - Enable
prefilter=TrueJudiciously: If your filter condition eliminates over 80% of records, pre-filtering avoids unnecessary vector distance calculations. Conversely, if your filter is very loose, post-filtering can sometimes scan raw vectors faster. - Effortless Backups: LanceDB organizes data as a directory of Lance data files and manifests. You can simply use
rcloneoraws s3 syncfor routine backups without needing database dumps or table locks.
