Three Approaches to Self-Hosting LLMs on Fedora Workstation
When running local LLMs smoothly on a personal PC or Fedora workstation, you typically face three architectural choices:
- Fully Native (Bare-metal): Download the Ollama binary to
/usr/local/binand set up Open WebUI usingvirtualenvoruv. This maximizes hardware performance, but cleaning it up later is painful. - 100% Containerized (Podman/Docker stack): Package both Ollama and WebUI into containers. However, configuring CDI (Container Device Interface) to pass through an NVIDIA GPU via rootless Podman on Fedora often runs into permission issues with
/dev/nvidia*. - Hybrid Model (Recommended): Install Ollama directly on the host OS to hook seamlessly into the CUDA driver, and run Open WebUI inside a Podman container to isolate the web environment.
Practical Evaluation of Each Approach
1. Fully Bare-Metal
Pros: Peak inference speeds. Zero latency overhead from namespaces or virtualization. Debugging CUDA driver issues is straightforward and fast.
Cons: Open WebUI pulls in dozens of Python and Node.js dependencies. Installing them directly on the host clutters your system environment, making future upgrades or clean removals time-consuming.
2. Fully Containerized
Pros: Clean lifecycle management. Tearing everything down is as simple as stopping the compose file.
Cons: Fedora runs SELinux in Enforcing mode by default. Granting a rootless container access to NVIDIA drivers frequently triggers permission denied errors unless you disable SELinux or craft custom policy rules.
3. The Hybrid Model (Native Ollama + Containerized Open WebUI)
This setup solves both problems completely. Ollama leverages the akmod-nvidia driver directly from RPM Fusion to offload 100% of layers onto vRAM, while the complex WebUI interface remains cleanly isolated in a single Podman container.
Why Fedora Workstations Should Use the Hybrid Model
After running Fedora as a daily driver workstation for over two years, I have seen how frequently its kernel updates. Every time a new kernel lands, akmods rebuilds the NVIDIA kernel modules.
Running Ollama natively allows the service to automatically pick up updated drivers without breaking container runtimes. You keep your OS clean while sidestepping all the headaches of configuring SELinux device passthrough.
Step-by-Step Deployment Guide
Step 1: Install the NVIDIA Driver from RPM Fusion
First, verify whether the system already recognizes your discrete GPU:
nvidia-smi
On a fresh install or if you are still running the open-source Nouveau driver, install the official driver packages via DNF:
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
# Wait for akmods to build kernel modules (usually 2-3 minutes), then reboot
sudo akmods --force
sudo reboot
Step 2: Install Ollama and Tweak the systemd Service
Download and install Ollama using the official script:
curl -fsSL https://ollama.com/install.sh | sh
By default, Ollama only listens locally on 127.0.0.1:11434. To optimize performance and ensure seamless access for WebUI, create a systemd override file:
sudo systemctl edit ollama.service
Paste the following configuration between the two commented lines:
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_ORIGINS=*"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KEEP_ALIVE=24h"
What these flags do in practice:
OLLAMA_FLASH_ATTENTION=1: Saves roughly 15% – 25% of vRAM on RTX 2000-series cards and newer when handling long context windows between 8k and 32k tokens.OLLAMA_KEEP_ALIVE=24h: Keeps models pinned in vRAM. This prevents the 4 – 6 second delay of reloading model weights from your NVMe SSD after periods of inactivity.
Reload the systemd daemon to apply changes:
sudo systemctl daemon-reload
sudo systemctl restart ollama
Step 3: Pull a Model and Verify CUDA Offloading
Pull a common model for testing—for example, qwen2.5:7b (the Q4_K_M quant consumes around 4.7GB of vRAM):
ollama run qwen2.5:7b
Open another terminal window and run nvidia-smi. If you see the ollama_llama_server process allocating vRAM and GPU utilization jumping to 90% – 100% during token generation, the GPU is working properly. On an RTX 3060 12GB, generation speeds typically land between 50-65 tokens/second.
Step 4: Launch Open WebUI with Podman
Fedora ships with Podman out of the box, so there is no need to install Docker. Create a persistent volume to preserve chat history across image updates:
podman volume create open-webui-data
podman run -d \
--name open-webui \
--network host \
-v open-webui-data:/app/backend/data:Z \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
--restart always \
ghcr.io/open-webui/open-webui:main
Two critical technical details to remember:
- The
:Zvolume flag: This is mandatory. It instructs Podman to automatically apply the proper SELinux label (container_file_t) to the volume, preventing silent Permission Denied errors on the SQLite database. - The
--network hostflag: Shares the host network namespace with the container. This allows Open WebUI to connect directly to Ollama at127.0.0.1:11434without configuring bridge networks or opening extra ports in firewalld.
Bước 5: Initial Account Setup
Navigate to http://localhost:8080 in your browser. The first account registered will automatically receive the Admin role. Head over to Settings > Models, where you will find all your downloaded models ready to go.
Performance Tuning Pro-Tips
- Running models that exceed vRAM: If you have an 8GB vRAM GPU (such as an RTX 3070 or 4060) and want to run a 14B model (which requires around 9GB vRAM), Ollama will automatically split layers between GPU vRAM and system RAM. It will run fine, though speed will drop to around 12-18 tokens/second.
- Changing the model storage location: By default, models are stored in
/usr/share/ollama/.ollama/modelson the root partition. If your root drive is running out of space, relocate them to a secondary NVMe drive by adding an environment variable to systemd:Environment="OLLAMA_MODELS=/mnt/nvme-data/ollama_models". - Disabling telemetry: For strict data privacy, append these two flags to your Podman run command:
-e SCARF_NO_ANALYTICS=true -e DO_NOT_TRACK=true.
