Why Do RAG Chatbots Keep Giving Irrelevant Answers?
Have you ever asked an internal chatbot a question only to receive an answer that completely misses the mark? The issue is rarely that the AI model lacks intelligence. The real culprit is usually document chunking.
Imagine holding a 50-page financial report. You take a pair of scissors, cut it into 200-word snippets, and toss them into a box. When your boss asks: "How much did retail revenue grow this quarter?", you happen to pull out a slip that reads simply: "This division achieved 18% year-over-year growth thanks to dealership expansion."
The slip bears no company name. No fiscal year. No one can tell whether "this division" refers to retail or wholesale. That is precisely how traditional RAG systems operate. Stripped of the source context, the LLM is left to guess or mix up entirely unrelated details.
How Does Contextual Retrieval Solve This Problem?
In September 2024, Anthropic introduced the Contextual Retrieval technique to address this root issue. The concept is straightforward: never store an isolated chunk of text on its own.
Before indexing data into a vector database, a compact model like Claude 3 Haiku reads the entire source document. The model then generates a concise explanatory summary (roughly 50 to 100 words) to prepend to each chunk.
As a result, the truncated slip from our earlier example becomes:
"This excerpt is from the Q3/2024 Financial Report of FPT Retail, under the FPT Shop business performance section: This division achieved 18% year-over-year growth…"
Suddenly, everything becomes clear. Both vector search (embeddings) and keyword search (BM25) can effortlessly surface this chunk. According to Anthropic’s benchmarks, combining Contextual Embeddings with BM25 reduces retrieval failure rates by 49%. When paired with a reranking step (such as Cohere Rerank), error rates drop by up to 67%.
Hands-on: Implementing Contextual Retrieval with Python
After testing this on an internal contract repository, retrieval accuracy improved significantly. Below is a complete implementation using the Anthropic SDK and ChromaDB.
Step 1: Set up the environment
pip install anthropic chromadb
Step 2: Create a context generation function with Prompt Caching
If a document spans dozens of pages and you resend the entire text for every single chunk, API costs will escalate quickly. The key here is Prompt Caching. By marking the source document for caching, Anthropic cuts read costs by up to 90% for subsequent chunks.
import os
from anthropic import Anthropic
client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
def generate_context_prefix(document_text: str, chunk_text: str) -> str:
"""
Use Claude 3 Haiku to generate 1-2 context sentences summarizing the chunk's placement.
Leverage Prompt Caching with cache_control to optimize cost and latency.
"""
response = client.messages.create(
model="claude-3-haiku-20240307",
max_tokens=150,
temperature=0.0,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": f"<document>\n{document_text}\n</document>",
# Cache the source document to reuse across chunks
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": (
f"Here is an excerpt from the document:\n<chunk>\n{chunk_text}\n</chunk>\n\n"
"Please provide a concise context (1-2 sentences) explaining where this chunk belongs "
"and what entity it discusses. Return only the context text, without any introductory remarks."
)
}
]
}
]
)
return response.content[0].text.strip()
Step 3: Prepend context and ingest into ChromaDB
Once you have the prefix, prepend it directly to the original chunk before generating vector embeddings.
import chromadb
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(name="kb_alpha")
document = """
ALPHA TECHNOLOGY JOINT STOCK COMPANY - ANNUAL REPORT 2024
Part 1: Business Overview
Alpha Company focuses on two core pillars: Cloud Computing and Artificial Intelligence (AI) Solutions.
Part 2: AI Division Results
This division recorded a 45% revenue growth in 2024, reaching 120 billion VND.
Part 3: 2025 Headcount Plan
Planning to hire 50 additional AI engineers at the Da Nang branch.
"""
# 2 chunks extracted from the source document
chunks = [
"This division recorded a 45% revenue growth in 2024, reaching 120 billion VND.",
"Planning to hire 50 additional AI engineers at the Da Nang branch."
]
for i, chunk in enumerate(chunks):
prefix = generate_context_prefix(document_text=document, chunk_text=chunk)
contextual_chunk = f"{prefix}\n\n{chunk}"
print(f"=== CHUNK {i+1} WITH CONTEXT ===")
print(contextual_chunk)
print("-" * 40)
collection.add(
documents=[contextual_chunk],
metadatas=[{"original_text": chunk, "source": "Bao_cao_Alpha_2024.pdf"}],
ids=[f"chunk_{i}"]
)
Step 4: Verify query results
Test it out with a realistic query:
query = "What was the revenue growth of Alpha's AI division?"
results = collection.query(query_texts=[query], n_results=1)
print("Best matching chunk found:")
print(results["documents"][0][0])
Without the prefix, the original snippet only contained the phrase "this division". Vector search could easily miss it because two critical keywords—"AI" and "Alpha"—were completely missing.
When Should You Use It?
A major advantage of this approach is that it does not require overhauling your system. You simply insert a data enrichment step into your ingestion pipeline.
- Real-world costs: Thanks to Anthropic’s Prompt Caching, processing 1 million input tokens with Claude 3 Haiku costs only around $0.30 (under 8,000 VND). That cost is negligible compared to the effort required to debug chatbot hallucinations.
- Recommended use cases: Particularly effective for commercial contracts, API documentation, medical records, and financial reports—documents packed with ambiguous pronouns and references like "Party A", "this clause", or "it".
- When you might not need it: If your knowledge base consists of concise Q&A items, standalone FAQ entries, or well-partitioned wiki articles with distinct ideas, traditional chunking will still work well.
