Posted inAI

Build Your Own ‘Google Search’ for AI: Combining SearXNG and LangChain to Break Knowledge Limits

Artificial Intelligence tutorial - IT technology blog
Artificial Intelligence tutorial - IT technology blog

LLM Barriers: When AI Doesn’t Know What’s Happening

After six months of implementing AI solutions, I’ve noticed a major issue: no matter how smart GPT-4 or Claude are, they remain stuck in the past (Knowledge Cutoff). If you ask about Bitcoin prices at 8 AM this morning or the latest changes in Next.js 15, the AI will either start “hallucinating” or refuse to answer.

To solve this, we need a RAG (Retrieval-Augmented Generation) mechanism on a global scale. Instead of being limited to a few PDF files, I’ll give the AI access to Google, Bing, and DuckDuckGo via SearXNG. This keeps the AI updated with the latest information while ensuring privacy.

Which Search API Should You Choose for Your AI Project?

I’ve experimented with several options, and here is a practical comparison for your consideration:

  • Google/Bing Search API: Quite expensive, around $5 per 1,000 queries. The registration process on Cloud Console is cumbersome, and your search data is collected by tech giants.
  • Tavily / Exa.ai: Specifically optimized for AI, so the results are very clean. However, the free tier only allows 1,000 requests/month. For production, the cost becomes a major headache.
  • SearXNG (The Optimal Choice): An open-source meta-search engine. You can self-host it on Docker, no API key is required, there are no search limits, and you have full control over your data.

Why SearXNG is the Perfect Match for LangChain?

The reason I chose SearXNG is its ability to return perfectly standardized JSON data. This allows LangChain to extract information quickly without needing complex custom code. It runs very smoothly when hosted on a private server.

The biggest plus is flexibility. If Google blocks an IP, SearXNG automatically switches to DuckDuckGo, Qwant, or Brave Search. Your system almost never experiences downtime.

Detailed Implementation Steps

Step 1: Setting up SearXNG with Docker

It takes about 2 minutes to set up your own search engine. First, create a docker-compose.yml file:

version: '3'

services:
  searxng:
    container_name: searxng
    image: searxng/searxng:latest
    ports:
      - "8080:8080"
    volumes:
      - ./searxng:/etc/searxng
    environment:
      - SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
    restart: always

In the settings.yml file, you need to enable JSON format so the AI can read it:

search:
    formats:
        - html
        - json
server:
    port: 8080
    bind_address: "0.0.0.0"
    secret_key: "enter_secure_key_here"

Run the command docker-compose up -d. Once you access localhost:8080 and see the search interface, you’re halfway there.

Step 2: Integrating into a LangChain Agent

LangChain comes with a built-in SearxSearchWrapper, making the connection very simple. Install the libraries with:

pip install langchain langchain-openai

Here is the Python code structure to create an AI Agent with real-time web search capabilities:

import os
from langchain_community.utilities import SearxSearchWrapper
from langchain_openai import ChatOpenAI
from langchain.agents import initialize_agent, Tool, AgentType

os.environ["SEARXNG_URL"] = "http://localhost:8080"

# Initialize the search engine
search = SearxSearchWrapper(searx_host=os.environ["SEARXNG_URL"])
tools = [
    Tool(
        name="Search",
        func=search.run,
        description="Use this when you need to answer questions about current events or the latest information on the web."
    )
]

# Use GPT-4 or replace with Ollama for a fully local setup
llm = ChatOpenAI(temperature=0, model="gpt-4-turbo")

agent = initialize_agent(
    tools, 
    llm, 
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, 
    verbose=True
)

# Use GPT-4 or replace with Ollama for a <a href="https://itfromzero.com/en/artificial-intelligence-en/localai-building-your-own-official-openai-api-server-on-linux.html">fully local setup</a>
response = agent.run("What is the SJC gold price today and how has it changed compared to yesterday?")
print(response)

Hard-Learned Lessons After 6 Months of Operation

To keep the system running stably in practice, keep these 4 key points in mind:

1. Control Response Times

Aggregating data from multiple sources can sometimes make SearXNG slow. You should set a timeout of about 10-15 seconds. Avoid letting the Agent fall into an infinite wait state if a specific search engine fails.

2. ‘Noise Filtering’ for Input Data

The internet is full of junk information. In the settings.yml file, prioritize high-quality sources like Reddit, Wikipedia, or major news sites. The cleaner the data, the less likely the AI is to experience hallucinations.

3. Strategies for Handling IP Blocking

If you query continuously, Google will blacklist your server IP. The solution is to use rotating proxies or simply disable the Google engine. Leverage DuckDuckGo and Brave Search as they are much more “lenient”.

4. Saving Tokens via Summarization

SearXNG results are often very long and can easily overflow the LLM’s Context Window. I usually use a secondary prompt to summarize search snippets to about 500-1000 words before feeding them into the final query. This reduces API costs by 40-60%.

Conclusion

Building your own AI Search Engine with SearXNG not only saves money but also provides absolute control. Your system now has “eyes” to observe the real world instead of relying solely on outdated memory. If you’re building a chatbot for business, this is definitely the most sustainable path.

Share:
Tags: