Posted inAI

Installing Microsoft GraphRAG with Ollama: Build Your Own 100% Local Knowledge Graph

Artificial Intelligence tutorial - IT technology blog
Artificial Intelligence tutorial - IT technology blog

Why Traditional RAG Sometimes Disappoints You

Have you ever built a Chatbot using RAG (Retrieval-Augmented Generation) only to find it getting “confused” when asked for a document overview? Standard RAG answers small details very well. However, when you ask, “What is the main content of these 500 pages of documentation?”, it often starts to struggle.

The problem lies in the vector similarity search mechanism. Traditional RAG finds text segments most similar to the query but lacks a big-picture view. Microsoft GraphRAG was created to fill that gap. Instead of just storing fragments, it builds a massive Knowledge Graph from your data.

Comparing Traditional RAG and GraphRAG

Here is why GraphRAG is becoming the new “weapon of choice” for AI projects requiring depth:

  • Traditional RAG (Baseline RAG):
    • Mechanism: Text chunking and vector search.
    • Pros: Fast response time, extremely easy deployment.
    • Cons: Struggles to understand connections between distant ideas.
  • GraphRAG (Microsoft):
    • Mechanism: Entity extraction, relationship mapping, and community building.
    • Pros: Excellent at answering “why” and “how” questions across the entire dataset.
    • Cons: Indexing takes 5-10 times longer and requires more computational resources.

Running Locally with Ollama: Benefits and Trade-offs

Microsoft designed GraphRAG with a preference for OpenAI. However, if you are working with sensitive corporate data, pushing everything to the Cloud is a major risk. Combining it with Ollama is the optimal way to keep your data strictly under your control.

Pros:

  • Security: Data never leaves your machine.
  • Cost-effective: Zero spending on OpenAI tokens. This is crucial because GraphRAG consumes a massive amount of tokens during indexing.
  • Flexibility: Easily switch from Llama 3 to Mistral or Phi-3 in a heartbeat.

Cons:

  • Hardware: You need at least a GPU with 12GB VRAM (like an RTX 3060) so that the indexing process doesn’t take forever.
  • Configuration: Requires minor tweaks to get Ollama and GraphRAG to “talk” to each other.

Real-world Experience

I tested this method on an internal technical manual over 200 pages long. The results showed that the system could coherently connect business logic from Chapter 1 to the final chapter. My boss was completely satisfied because no data was leaked. If your machine has 32GB of RAM and a decent graphics card, this is the most worthwhile direction to invest in right now.

Detailed Deployment Guide

Step 1: Prepare Ollama

Download Ollama at ollama.com. Then, pull two important models. One model is for reasoning and the other is specialized for creating vectors (Embedding).

# Main model for language processing
ollama pull llama3

# Specialized model for vector generation
ollama pull nomic-embed-text

Step 2: Set up GraphRAG Environment

Use Python 3.10 or higher. I recommend using a virtual environment to keep your system clean.

# Create and activate virtual environment
python -m venv graphrag-env
source graphrag-env/bin/activate # Windows: graphrag-env\Scripts\activate

# Install main library
pip install graphrag

Step 3: Initialize the Project

Create a working directory and have GraphRAG generate sample configuration files:

mkdir my-graphrag-project
cd my-graphrag-project
mkdir input

# Initialize structure
python -m graphrag.index --init --root .

Step 4: Configure settings.yaml

This is the most important part to “force” GraphRAG to communicate with Ollama. Open the settings.yaml file and update the following parameters:

encoding_model: cl100k_base
llm:
  type: openai_chat
  api_base: http://localhost:11434/v1
  api_key: ollama # Enter dummy key to pass validation
  model: llama3
  model_supports_json: true # Crucial for accurate entity extraction

embeddings:
  async_mode: threaded
  llm:
    type: openai_embedding
    api_base: http://localhost:11434/api # Dedicated endpoint for embedding
    api_key: ollama
    model: nomic-embed-text

Step 5: Run Indexing

Drop your .txt files into the input/ folder. Then, start the knowledge graph construction process:

python -m graphrag.index --root .

Note: If you have 50 text files, this process can take anywhere from 30 minutes to an hour depending on your GPU power. Don’t get impatient if you see the terminal cycling through workflows continuously.

Step 6: Q&A with Your Data

Once indexing is complete, you have two excellent query modes:

Global Search: Use for questions that require synthesizing the entire document.

python -m graphrag.query --root . --method global "Summarize the 3 main points of the document"

Local Search: Use to dive deep into a specific character or event.

python -m graphrag.query --root . --method local "What is the relationship between A and B?"

Small Tips for Performance Optimization

During the process, I noticed that Llama 3 8B sometimes extracts entities a bit excessively. If you own a “beast” graphics card (24GB VRAM or more), go ahead and use Llama 3 70B. The results will be much more accurate and sharper.

Another tip: If your documents are highly specialized (like medical or legal), reduce the chunk_size in the settings file to about 600. This helps the model avoid missing complex relationships between short paragraphs.

Running offline might seem like a bit of configuration effort at first. However, the feeling of fully mastering a powerful AI system right on your personal laptop is truly worth it. Good luck with your installation!

Share:
Tags: