Build Your Own Local AI Chatbot: Deploying Ollama and Open WebUI with Docker Compose – ITFROMZERO

Table of Contents

The 2 AM Data Leak and the Decision to Move AI In-house

It all started when a dev on my team accidentally pasted a code snippet containing an AWS API Key into ChatGPT to ask for a logic fix. Just 15 minutes later, the security department sounded the red alert. Using “off-the-shelf” solutions like ChatGPT or Claude is convenient, but the trade-off is that you have zero control over your input data. At that moment, I realized that deploying a Local LLM (Large Language Model) was no longer just a hobby for tinkering—it was a mandatory security requirement.

In the production system of over 30 containers I manage, isolation is the top priority. I chose the duo of Ollama and Open WebUI running on Docker Compose. This solution saves about 40% in maintenance effort compared to a native installation. You can port the entire system from your local machine to a server in just minutes.

Why You Shouldn’t Install Directly (Native)?

Before settling on the Docker approach, I tried a few other methods and learned some hard lessons:

Native Installation (Binary): Very fast but extremely prone to CUDA or Python library conflicts. A single OS update could break the entire AI engine instantly.
Private Cloud (AWS Bedrock): Good security but very high maintenance costs. With continuous query demands, the monthly bill can reach thousands of dollars.
Docker Compose: This is the optimal choice. Everything is neatly packaged. Want to switch from NVIDIA GPU to CPU or back up your data? You only need to interact with the YAML file and copy the data folder.

The Power of the Ollama & Open WebUI Duo

Ollama acts as the “heart” controlling models like Llama 3.1, Mistral, or Qwen. Meanwhile, Open WebUI provides a smooth interface, supporting both RAG (Retrieval-Augmented Generation) and professional user management. This combination delivers an experience on par with the paid version of ChatGPT.

Minimum Hardware Requirements

To ensure the chatbot responds quickly without lag, you need to prepare:

Docker & Docker Compose: Installed and running stably.
NVIDIA GPU: Minimum 8GB VRAM recommended (such as an RTX 3060 or 4060). With 8GB VRAM, you can smoothly run 7B-8B models at speeds of 50-70 tokens/s.
NVIDIA Container Toolkit: A mandatory component for Docker to communicate with the graphics card driver.

# Command to check GPU driver
nvidia-smi

Quick Deployment with Docker Compose

Don’t run individual docker run commands manually as they are difficult to manage in terms of networking. Instead, bundle everything into a docker-compose.yaml file for centralized management.

Create a local-ai directory and paste the following content into the configuration file:

services:
  ollama:
    volumes:
      - ./ollama:/root/.ollama
    container_name: ollama
    pull_policy: always
    tty: true
    restart: unless-stopped
    image: ollama/ollama:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    volumes:
      - ./open-webui:/app/backend/data
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - 'OLLAMA_BASE_URL=http://ollama:11434'
      - 'WEBUI_SECRET_KEY=change_this_key_immediately'
    restart: unless-stopped

Decoding Key Parameters

Volume: Mounting the ./ollama folder to the host machine helps retain models (which are often 5GB – 40GB) when you upgrade containers.
Deploy (GPU): This section enables hardware power. If your machine only has a CPU, delete the entire deploy block to avoid startup errors.
OLLAMA_BASE_URL: Uses the service name ollama instead of an IP address. Docker will automatically route internal traffic between the two containers.

Startup and Experience

Just a single command is needed to activate the entire system:

docker compose up -d

After Docker finishes pulling the images, access http://localhost:3000. In the Open WebUI interface, follow these steps:

Register an Admin account (this data resides entirely on your machine).
Go to Settings > Models, and enter llama3.1:8b.
Click Pull. With a 100Mbps connection, this process takes about 3-5 minutes for 4.7GB of data.

Real-world Troubleshooting

During operation, I often encounter these three main issues:

1. GPU Driver Not Found Error

If you see a log reporting could not select device driver, it means you haven’t installed the NVIDIA Container Toolkit. Run the following command on Ubuntu to fix it:

sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

2. Chatbot Responding Extremely Slowly

Check docker logs -f ollama. If you see 100% CPU load but the GPU is idle, your NVIDIA driver is likely too old. Ollama will automatically fallback to CPU mode, causing speeds to drop from 60 tokens/s to just 2-3 tokens/s.

3. Out of Memory (RAM Overflow)

An 8B model needs about 5GB of VRAM to operate stably. If you try to run a 70B model on an RTX 3060, the system will crash immediately. The solution is to prioritize using Quantized (GGUF) versions to reduce memory usage while maintaining intelligence.

Conclusion

Taking control of your AI system allows you to confidently handle sensitive client data without fear of leaks. Additionally, operating costs are nearly zero after the initial hardware investment. If you plan to deploy this for a company, remember to configure an Nginx Reverse Proxy and enable SSL to protect the connection between workstations and the server.