Real-World Context & Limitations of Full-Text Search
At 2:00 AM, PagerDuty triggered a critical alert. The internal company wiki and log lookup system were responding abnormally slowly, with the timeout rate hitting 42%. After SSHing into the production server (8 vCPUs, 16GB RAM), we found that the LIKE '%keyword%' queries and the Elasticsearch cluster were eating up 98% of RAM while scanning through over 250,000 unstructured records.
The biggest bottleneck was the search experience itself. When a user typed ‘cannot ping gateway’, the relational database only searched for that exact phrase. The system completely overlooked relevant documents containing synonyms like ‘network connection dropped’ or ‘packet loss’.
As the system kept growing, forcing a relational database to shoulder the entire text search workload was an impending disaster for the backend. Setting up a massive cluster like Milvus or paying monthly fees for Pinecone was overkill for internal needs. This is where a lightweight vector database running directly inside the Python process—such as ChromaDB—comes into play.
ChromaDB tackles the core problem directly: converting text into vector embeddings (arrays of floating-point numbers representing semantics) and querying them via the HNSW algorithm with a latency of just 12–25ms on a 50,000-record dataset.
Installation & SQLite Compatibility Pitfalls
First, create an isolated virtual environment on Ubuntu 22.04 LTS to decouple dependencies for OnnxRuntime and PyTorch.
# Initialize and activate the virtual environment
python3 -m venv venv
source venv/bin/activate
# Install chromadb and sentence-transformers
pip install --upgrade pip
pip install chromadb sentence-transformersA classic issue when deploying to older servers (CentOS 7, Ubuntu 18.04, or Debian 10): ChromaDB requires SQLite >= 3.35.0. Otherwise, the script will immediately throw RuntimeError: Your system has an unsupported version of sqlite3.
To patch this quickly without recompiling SQLite from source, install the binary package:
pip install pysqlite3-binaryInsert the following hook at the very top of your entry file, before any other imports:
__import__('pysqlite3')
import sys
sys.modules['sqlite3'] = sys.modules.pop('pysqlite3')
import chromadbDetailed Production Configuration
By default, ChromaDB keeps data in-memory. If the service restarts, the entire index is lost. Therefore, using PersistentClient to persist data to disk is mandatory.
1. Initializing Persistent Client & Choosing an Embedding Model
For Vietnamese or multilingual documents, ChromaDB’s default model (all-MiniLM-L6-v2) yields sub-optimal results. You should switch to paraphrase-multilingual-MiniLM-L12-v2 so the engine can accurately capture Vietnamese semantics.
import chromadb
from chromadb.utils import embedding_functions
# Initialize persistent storage in a local directory
client = chromadb.PersistentClient(path="./chroma_storage")
# Declare a multilingual model optimized for Vietnamese
multilingual_ef = embedding_functions.SentenceTransformerEmbeddingFunction(
model_name="paraphrase-multilingual-MiniLM-L12-v2"
)
# Create a collection using Cosine distance
collection = client.get_or_create_collection(
name="incident_knowledgebase",
embedding_function=multilingual_ef,
metadata={"hnsw:space": "cosine"}
)2. Indexing and Attaching Metadata
When data scales to tens of thousands of records, attaching metadata enables fast pre-filtering (metadata filtering) before computing vector similarities, significantly reducing CPU overhead.
# Real-world incident logs
documents = [
"Server lost network connection due to gateway configuration conflict in Netplan",
"Nginx service returned 502 Bad Gateway because Gunicorn worker was overloaded",
"PostgreSQL rejected connections after reaching max_connections limit (500/500)",
"Partition /var/log reached 100% disk space causing processes to crash"
]
metadatas = [
{"service": "network", "severity": "critical", "ticket_id": 2041},
{"service": "web_proxy", "severity": "high", "ticket_id": 2042},
{"service": "database", "severity": "critical", "ticket_id": 2043},
{"service": "system", "severity": "medium", "ticket_id": 2044}
]
ids = ["inc_2041", "inc_2042", "inc_2043", "inc_2044"]
# Use upsert instead of add to avoid duplicate ID errors
collection.upsert(
documents=documents,
metadatas=metadatas,
ids=ids
)
print(f"Total documents currently in collection: {collection.count()}")3. Querying Semantic Search with Filtering
Let’s run a test query: ‘cannot access web server’. Notice that the exact phrase ‘cannot access’ does not appear in the original documents.
query_text = "cannot access web server"
# Find the 2 nearest neighbors with high or critical severity
results = collection.query(
query_texts=[query_text],
n_results=2,
where={"severity": {"$in": ["high", "critical"]}}
)
for idx in range(len(results['documents'][0])):
doc = results['documents'][0][idx]
distance = results['distances'][0][idx]
meta = results['metadatas'][0][idx]
doc_id = results['ids'][0][idx]
print(f"Top {idx+1} [ID: {doc_id}] - Cosine Distance: {distance:.4f}")
print(f"Content: {doc}")
print(f"Metadata: {meta}")
print("-" * 45)Benchmarking & Production Operations
Passing tests in development is one thing; keeping the system stable under heavy production load is another challenge.
1. Monitoring Disk Footprint
Data is stored in an SQLite database (holding metadata) alongside a binary directory storing HNSW indexes. Regularly check the directory size:
# Check the index directory size
du -sh ./chroma_storage
# Verify the actual number of synced records in SQLite
sqlite3 ./chroma_storage/chroma.sqlite3 "SELECT count(*) FROM collections;"2. Benchmarking Query Latency
It is important to differentiate: latency spikes usually stem from CPU embedding inference, not the vector search itself.
import time
start = time.perf_counter()
res = collection.query(query_texts=["disk full of logs"], n_results=1)
elapsed_ms = (time.perf_counter() - start) * 1000
print(f"Query latency: {elapsed_ms:.2f} ms")
# If this latency exceeds 80ms on a 4 vCPU server, offload embedding inference to a GPU worker or leverage quantization.3. Three Core Production Best Practices
- RAM Consumption: The multilingual model consumes around 450MB–600MB of RAM. On a 1GB–2GB RAM VPS, never run the embedding process within the main web application process. Decouple it into a standalone service via client-server mode:
chroma run --path ./chroma_storage --port 8000. - Backup Strategy: The
./chroma_storagedirectory contains both SQLite files and HNSW binary indexes. Before running rsync or taking disk snapshots, ensure no active write operations are in flight to prevent index corruption or out-of-sync states. - Batch Size for Data Ingestion: When importing more than 10,000 records, split them into batches of 250 to 500 documents. Passing an oversized payload in a single upsert call can trigger Out of Memory (OOM) errors due to memory spikes during the vectorization phase.
