Build Your Own AI Assistant for Obsidian with Khoj on Linux: Offline, Private, and Free

When your note vault becomes a data “graveyard”

After more than 5 years working in IT, I’ve accumulated over 3,000 Markdown files in Obsidian. From Python code snippets to complex system design documents, I saved everything. However, finding old information became a nightmare. Obsidian’s keyword-based search often returns dozens of irrelevant results, making me waste hours just to find a solution I had already written.

I tried copying data into ChatGPT to ask questions, but this seriously violated company security policies. After 6 months of using Khoj on Linux, I realized this is the most practical solution. It “revives” those old notes into a virtual assistant that understands everything you’ve ever written, operating 100% offline.

Comparing AI approaches for note-taking

Here is a quick comparison table to show why Khoj is worth installing on a Linux environment.

Why I chose Khoj over building a custom RAG with LangChain?

Many developers might think: “Why not just use Ollama + LangChain + ChromaDB?” In fact, I tried that. However, maintaining a pipeline to sync new files is exhausting. You also have to handle Markdown formatting and build a chat interface yourself.

Khoj handles everything automatically:

• Automatic Indexing: Updates the vector database as soon as you save a file.
• Multi-platform support: Available as an Obsidian plugin, web interface, and desktop app.
• Flexibility: Can run both local models (Ollama) and external APIs if your hardware is limited.

Deploying Khoj on Linux using Docker

Installing via Docker is the cleanest way to avoid Python library conflicts. I performed this on Ubuntu 22.04, but other distros like Fedora or Arch follow a similar process.

Step 1: Preparation

Check if Docker is installed. If you are using an NVIDIA graphics card, install the nvidia-container-toolkit to make the AI respond 5-10 times faster than using a CPU.

Step 2: Docker Compose Configuration

Create a workspace and configuration file:

mkdir ~/khoj-ai && cd ~/khoj-ai
nano docker-compose.yml

Paste the following content into the file. Make sure to replace the path with your actual Obsidian vault directory:

version: '3.8'
services:
  khoj:
    image: khoj/khoj:latest
    container_name: khoj-server
    volumes:
      - ./khoj_data:/home/khoj/.khoj
      - /home/user/Documents/MyObsidianVault:/home/khoj/data
    ports:
      - "42110:42110"
    restart: unless-stopped
    environment:
      - KHOJ_DOMAIN=http://localhost:42110
      - OFFLINE=true

Step 3: Launching

Start the server with the following command:

docker compose up -d

Access http://localhost:42110 to create an admin account. Since it’s running offline, this information stays only on your hard drive.

Turning Obsidian into a smart assistant

This is where the magic happens. You can chat directly with your data within the editor interface.

1. Go to Settings > Community Plugins in Obsidian and install the “Khoj” plugin.
2. In the plugin settings, select Self-hosted.
3. Enter the URL: http://localhost:42110.
4. Click Index to begin the data scanning process.

Real-world experience: With about 2,000 files, my machine (Core i7, 16GB RAM) took 10 minutes to finish indexing. During this time, the CPU will spike to around 80-90%; just let it run and don’t interrupt it.

Connecting Ollama for 100% Offline Chat

To get high-quality answers like ChatGPT without an internet connection, use Ollama. In the Khoj Web UI, go to Settings > Models and add the following configuration:

• Model Name: llama3 (or mistral)
• API Endpoint: http://host.docker.internal:11434/v1

Real-world evaluation after six months

Pros

• Semantic Search: I asked “How to fix Nginx 502 error from last year,” and Khoj found the exact note even though the filename was messy.
• Security: All sensitive project source code is indexed locally. Not a single byte of data is sent to the cloud.

Cons

• Resource Intensive: Running both Khoj and Llama 3 consumes about 8-10GB of RAM. You should have at least 16GB of RAM for smooth operation.
• Speed: Without a GPU, the AI response is a bit slow, at about 2-4 words per second.

Khoj is not just a search tool. It is a partner that understands your thinking through what you have written. If you prioritize privacy and use Linux, try installing it today.