A Guide to Implementing Corrective RAG (CRAG) with Python: Automated Document Evaluation and Web Search

Context & Why You Need It

Asking an internal bot a policy question and receiving a “confidently wrong” answer is an all-too-familiar experience for many teams. With traditional RAG, the workflow is strictly linear: a question comes in, vector search retrieves the top 3–5 most similar text chunks, and they are passed straight to the LLM.

Vector search only captures surface-level semantic similarity; it has no awareness of whether a document actually answers the question. For instance, suppose an employee asks about a new remote work policy introduced last week. If the knowledge base only contains last year’s draft, the bot will still pull it as context and generate an inaccurate response.

Here is an analogy: traditional RAG is like a front-desk clerk quickly grabbing a few old sheets of paper from a drawer and reading them directly to a customer. Corrective RAG (CRAG) adds a verification step to that process: a quick glance to see if the papers are actually relevant. If the documents are accurate, the bot uses them right away. If the information is irrelevant or missing entirely, the system automatically opens a browser to search trusted external sources.

In practice, when deployed for an internal Q&A system serving 800 employees, CRAG reduced the hallucination rate from 27% down to 4.5%. The engineering team didn’t need to rush into fine-tuning models or re-indexing the entire document repository.

Installation

LangGraph is an ideal choice for managing branching logic and loops in CRAG, paired with Tavily Search, which is specifically optimized for agentic web queries.

Install the required packages directly from your terminal:

pip install langchain langchain-openai langchain-community langgraph tavily-python chromadb tiktoken

Export your API keys for OpenAI and Tavily into your environment (Tavily offers a free tier of 1,000 requests/month):

export OPENAI_API_KEY="sk-..."
export TAVILY_API_KEY="tvly-..."

Detailed Configuration

A standard CRAG architecture operates across 4 stages:

  1. Retrieve: Pull 2–3 document chunks from the internal vector database.
  2. Grade Documents: Use a lightweight LLM to assign a binary score (yes/no) evaluating whether the documents are relevant to the question.
  3. Web Search Fallback: Automatically trigger Tavily Search whenever internal documents are deemed irrelevant or empty.
  4. Generate: Synthesize the filtered data to generate a coherent answer for the user.

1. Initializing a Mock Vector Store

Create a file named crag_pipeline.py and pre-load some mock company policies as sample data:

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_core.documents import Document

# Mock internal data
docs = [
    Document(page_content="The company's remote work policy allows a maximum of 2 days per week for employees who have passed probation."),
    Document(page_content="Salaries are paid regularly on the 5th of each month via Vietcombank bank accounts."),
    Document(page_content="The annual tech certification training budget is up to 15 million VND per technical employee.")
]

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(documents=docs, embedding=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

2. Building the Document Grader

The core of CRAG lies in document verification. Instead of letting the LLM generate a verbose answer, we enforce a structured binary response: yes (relevant) or no (irrelevant).

from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

class GradeDocuments(BaseModel):
    binary_score: str = Field(description="Whether the document is relevant to the question, 'yes' or 'no'")

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
structured_llm_grader = llm.with_structured_output(GradeDocuments)

system_prompt = """You are a document quality assessment specialist.
Your task is to determine whether the provided document contains information that answers the user's question.
Only evaluate as 'yes' if the document is genuinely relevant; otherwise, evaluate as 'no'."""

grade_prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "Retrieved document:\n\n{document}\n\nUser question: {question}")
])

doc_grader = grade_prompt | structured_llm_grader

3. Defining Nodes and State in LangGraph

LangGraph manages data flow through a TypedDict called GraphState. Each node receives the state, executes its specific logic, and updates the state accordingly.

from typing import List
from typing_extensions import TypedDict
from langchain_community.tools.tavily_search import TavilySearchResults

class GraphState(TypedDict):
    question: str
    generation: str
    web_search_needed: bool
    documents: List[Document]

def retrieve(state):
    question = state["question"]
    documents = retriever.invoke(question)
    return {"documents": documents, "question": question, "web_search_needed": False}

def grade_documents(state):
    question = state["question"]
    documents = state["documents"]
    
    filtered_docs = []
    web_search_needed = False
    
    for d in documents:
        score = doc_grader.invoke({"question": question, "document": d.page_content})
        if score.binary_score == "yes":
            filtered_docs.append(d)
        else:
            web_search_needed = True
            
    if len(filtered_docs) == 0:
        web_search_needed = True
        
    return {"documents": filtered_docs, "question": question, "web_search_needed": web_search_needed}

def web_search(state):
    question = state["question"]
    documents = state.get("documents", [])
    
    web_tool = TavilySearchResults(k=2)
    web_results = web_tool.invoke({"query": question})
    
    web_content = "\n".join([d["content"] for d in web_results])
    documents.append(Document(page_content=web_content, metadata={"source": "web_search"}))
    
    return {"documents": documents, "question": question}

def generate(state):
    question = state["question"]
    documents = state["documents"]
    
    context = "\n\n".join([d.page_content for d in documents])
    gen_prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a virtual assistant answering inquiries. Answer the question based on the following context:\n\n{context}"),
        ("human", "{question}")
    ])
    
    rag_chain = gen_prompt | llm
    response = rag_chain.invoke({"context": context, "question": question})
    return {"generation": response.content}

4. Assembling the Complete Workflow

Once the individual nodes are defined, we connect them into a graph with conditional branching:

from langgraph.graph import END, StateGraph

workflow = StateGraph(GraphState)

# Add nodes
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade_documents", grade_documents)
workflow.add_node("web_search", web_search)
workflow.add_node("generate", generate)

# Configure execution flow
workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade_documents")

def decide_to_generate(state):
    if state["web_search_needed"]:
        return "web_search"
    return "generate"

workflow.add_conditional_edges(
    "grade_documents",
    decide_to_generate,
    {
        "web_search": "web_search",
        "generate": "generate"
    }
)
workflow.add_edge("web_search", "generate")
workflow.add_edge("generate", END)

app = workflow.compile()

Testing & Monitoring

Test the workflow across two real-world scenarios to observe the branching behavior:

# Scenario 1: Question can be answered using internal data
print("--- TEST 1: INTERNAL DATA ---")
res_1 = app.invoke({"question": "How many days per week are employees allowed to work remotely at most?"})
print("Answer:", res_1["generation"])

# Scenario 2: Information is completely absent internally, requiring external web search
print("\n--- TEST 2: WEB SEARCH REQUIRED ---")
res_2 = app.invoke({"question": "How is Apple's stock price performing today?"})
print("Answer:", res_2["generation"])

Key Metrics to Monitor in Production

Here are 3 critical metrics to track on your dashboard (such as LangSmith or OpenTelemetry) when running in production:

  • Web Fallback Rate: A healthy baseline typically stays below 15–20%. If this rate exceeds 35%, your internal knowledge base has significant gaps. Update your documentation instead of incurring constant API costs from web searches.
  • End-to-End Latency: CRAG introduces an extra LLM call for document grading and may trigger a web search. The pipeline generally adds 1.2 to 2.5 seconds compared to vanilla RAG. Optimization tip: use gpt-4o-mini (latency ~300ms) or a quantized local 7B model for the grading stage.
  • Token & API Costs: The grader evaluates each retrieved chunk. You should fix the retriever parameter to k=2 or k=3. Keeping context concise can save up to 40% in token costs compared to stuffing numerous unvetted documents into the final generation prompt.
Share: