Context & Why You Need It
Asking an internal bot a policy question and receiving a “confidently wrong” answer is an all-too-familiar experience for many teams. With traditional RAG, the workflow is strictly linear: a question comes in, vector search retrieves the top 3–5 most similar text chunks, and they are passed straight to the LLM.
Vector search only captures surface-level semantic similarity; it has no awareness of whether a document actually answers the question. For instance, suppose an employee asks about a new remote work policy introduced last week. If the knowledge base only contains last year’s draft, the bot will still pull it as context and generate an inaccurate response.
Here is an analogy: traditional RAG is like a front-desk clerk quickly grabbing a few old sheets of paper from a drawer and reading them directly to a customer. Corrective RAG (CRAG) adds a verification step to that process: a quick glance to see if the papers are actually relevant. If the documents are accurate, the bot uses them right away. If the information is irrelevant or missing entirely, the system automatically opens a browser to search trusted external sources.
In practice, when deployed for an internal Q&A system serving 800 employees, CRAG reduced the hallucination rate from 27% down to 4.5%. The engineering team didn’t need to rush into fine-tuning models or re-indexing the entire document repository.
Installation
LangGraph is an ideal choice for managing branching logic and loops in CRAG, paired with Tavily Search, which is specifically optimized for agentic web queries.
Install the required packages directly from your terminal:
pip install langchain langchain-openai langchain-community langgraph tavily-python chromadb tiktoken
Export your API keys for OpenAI and Tavily into your environment (Tavily offers a free tier of 1,000 requests/month):
export OPENAI_API_KEY="sk-..."
export TAVILY_API_KEY="tvly-..."
Detailed Configuration
A standard CRAG architecture operates across 4 stages:
- Retrieve: Pull 2–3 document chunks from the internal vector database.
- Grade Documents: Use a lightweight LLM to assign a binary score (yes/no) evaluating whether the documents are relevant to the question.
- Web Search Fallback: Automatically trigger Tavily Search whenever internal documents are deemed irrelevant or empty.
- Generate: Synthesize the filtered data to generate a coherent answer for the user.
1. Initializing a Mock Vector Store
Create a file named crag_pipeline.py and pre-load some mock company policies as sample data:
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_core.documents import Document
# Mock internal data
docs = [
Document(page_content="The company's remote work policy allows a maximum of 2 days per week for employees who have passed probation."),
Document(page_content="Salaries are paid regularly on the 5th of each month via Vietcombank bank accounts."),
Document(page_content="The annual tech certification training budget is up to 15 million VND per technical employee.")
]
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(documents=docs, embedding=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})
2. Building the Document Grader
The core of CRAG lies in document verification. Instead of letting the LLM generate a verbose answer, we enforce a structured binary response: yes (relevant) or no (irrelevant).
from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
class GradeDocuments(BaseModel):
binary_score: str = Field(description="Whether the document is relevant to the question, 'yes' or 'no'")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
structured_llm_grader = llm.with_structured_output(GradeDocuments)
system_prompt = """You are a document quality assessment specialist.
Your task is to determine whether the provided document contains information that answers the user's question.
Only evaluate as 'yes' if the document is genuinely relevant; otherwise, evaluate as 'no'."""
grade_prompt = ChatPromptTemplate.from_messages([
("system", system_prompt),
("human", "Retrieved document:\n\n{document}\n\nUser question: {question}")
])
doc_grader = grade_prompt | structured_llm_grader
3. Defining Nodes and State in LangGraph
LangGraph manages data flow through a TypedDict called GraphState. Each node receives the state, executes its specific logic, and updates the state accordingly.
from typing import List
from typing_extensions import TypedDict
from langchain_community.tools.tavily_search import TavilySearchResults
class GraphState(TypedDict):
question: str
generation: str
web_search_needed: bool
documents: List[Document]
def retrieve(state):
question = state["question"]
documents = retriever.invoke(question)
return {"documents": documents, "question": question, "web_search_needed": False}
def grade_documents(state):
question = state["question"]
documents = state["documents"]
filtered_docs = []
web_search_needed = False
for d in documents:
score = doc_grader.invoke({"question": question, "document": d.page_content})
if score.binary_score == "yes":
filtered_docs.append(d)
else:
web_search_needed = True
if len(filtered_docs) == 0:
web_search_needed = True
return {"documents": filtered_docs, "question": question, "web_search_needed": web_search_needed}
def web_search(state):
question = state["question"]
documents = state.get("documents", [])
web_tool = TavilySearchResults(k=2)
web_results = web_tool.invoke({"query": question})
web_content = "\n".join([d["content"] for d in web_results])
documents.append(Document(page_content=web_content, metadata={"source": "web_search"}))
return {"documents": documents, "question": question}
def generate(state):
question = state["question"]
documents = state["documents"]
context = "\n\n".join([d.page_content for d in documents])
gen_prompt = ChatPromptTemplate.from_messages([
("system", "You are a virtual assistant answering inquiries. Answer the question based on the following context:\n\n{context}"),
("human", "{question}")
])
rag_chain = gen_prompt | llm
response = rag_chain.invoke({"context": context, "question": question})
return {"generation": response.content}
4. Assembling the Complete Workflow
Once the individual nodes are defined, we connect them into a graph with conditional branching:
from langgraph.graph import END, StateGraph
workflow = StateGraph(GraphState)
# Add nodes
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade_documents", grade_documents)
workflow.add_node("web_search", web_search)
workflow.add_node("generate", generate)
# Configure execution flow
workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade_documents")
def decide_to_generate(state):
if state["web_search_needed"]:
return "web_search"
return "generate"
workflow.add_conditional_edges(
"grade_documents",
decide_to_generate,
{
"web_search": "web_search",
"generate": "generate"
}
)
workflow.add_edge("web_search", "generate")
workflow.add_edge("generate", END)
app = workflow.compile()
Testing & Monitoring
Test the workflow across two real-world scenarios to observe the branching behavior:
# Scenario 1: Question can be answered using internal data
print("--- TEST 1: INTERNAL DATA ---")
res_1 = app.invoke({"question": "How many days per week are employees allowed to work remotely at most?"})
print("Answer:", res_1["generation"])
# Scenario 2: Information is completely absent internally, requiring external web search
print("\n--- TEST 2: WEB SEARCH REQUIRED ---")
res_2 = app.invoke({"question": "How is Apple's stock price performing today?"})
print("Answer:", res_2["generation"])
Key Metrics to Monitor in Production
Here are 3 critical metrics to track on your dashboard (such as LangSmith or OpenTelemetry) when running in production:
- Web Fallback Rate: A healthy baseline typically stays below 15–20%. If this rate exceeds 35%, your internal knowledge base has significant gaps. Update your documentation instead of incurring constant API costs from web searches.
- End-to-End Latency: CRAG introduces an extra LLM call for document grading and may trigger a web search. The pipeline generally adds 1.2 to 2.5 seconds compared to vanilla RAG. Optimization tip: use
gpt-4o-mini(latency ~300ms) or a quantized local 7B model for the grading stage. - Token & API Costs: The grader evaluates each retrieved chunk. You should fix the retriever parameter to
k=2ork=3. Keeping context concise can save up to 40% in token costs compared to stuffing numerous unvetted documents into the final generation prompt.
