Why Self-Host Local LLMs Instead of Relying on Cloud APIs?
In enterprise infrastructure environments, data security is always the biggest hurdle. Sending log files containing internal IP addresses, access tokens, or database dumps to cloud APIs poses an exceptionally high risk of data leakage. Beyond compliance concerns, token costs can skyrocket when you need to analyze tens of thousands of log lines every day.
Self-hosting open-source models with Ollama solves both of these challenges completely. Your data never leaves your server, and API costs drop to zero. Combined with the official Ollama Python SDK, you can embed AI directly into your automation scripts with just a few familiar lines of code.
Environment Setup
First, make sure the Ollama service is installed and listening stably on the default port 11434.
1. Install Ollama Server
Run the following command on your Ubuntu or macOS server to install the latest version:
curl -fsSL https://ollama.com/install.sh | sh
Next, pull a compact model to test. The llama3.2:3b model is only about 2.0GB in size and consumes under 2.5GB of VRAM, while delivering very consistent inference in both English and Vietnamese:
ollama pull llama3.2:3b
ollama list
2. Install the Python SDK
It is recommended to create a dedicated virtual environment to prevent dependency conflicts:
python3 -m venv .venv
source .venv/bin/activate
pip install ollama
Hands-On with the Ollama Python SDK: From Basic to Advanced
The SDK provides two primary interfaces: synchronous for straightforward tasks and asynchronous (async) for high-throughput backends.
1. Text Generation and Real-Time Streaming
If a prompt requires a lengthy response, waiting for the model to generate the complete text can feel sluggish. Enabling the stream=True parameter reduces Time-To-First-Token (TTFT) to under 300ms, displaying words as soon as they are generated.
import ollama
# Call the model with streaming enabled
stream = ollama.generate(
model='llama3.2:3b',
prompt='Briefly list 3 best practices for writing secure Dockerfiles:',
stream=True
)
print("Response from local LLM:")
for chunk in stream:
print(chunk['response'], end='', flush=True)
print()
2. Multi-Turn Conversation Management (Chat API)
Unlike generate, which only takes a raw prompt, the chat function preserves conversation flow using a list of role-based messages (system, user, assistant).
from ollama import Client
# Initialize client pointing to local Ollama instance
client = Client(host='http://localhost:11434')
messages = [
{
'role': 'system',
'content': 'You are a DevOps engineer. Provide concise responses focused directly on technical solutions.'
},
{
'role': 'user',
'content': 'How can I reduce the Docker image size for a Python application?'
}
]
# First conversation turn
response = client.chat(model='llama3.2:3b', messages=messages)
print(response['message']['content'])
# Append response to chat history to maintain context
messages.append(response['message'])
messages.append({
'role': 'user',
'content': 'Besides Alpine, are there other effective techniques?'
})
# Second conversation turn
follow_up = client.chat(model='llama3.2:3b', messages=messages)
print(follow_up['message']['content'])
Note: By default, Ollama sets the context window (num_ctx) to 2048 tokens. For extended conversations or large document ingestion, you should increase this parameter in the options dictionary: options={'num_ctx': 4096}.
3. Enforcing Structured JSON Output
When building log parsing scripts, we need structured data conforming to a schema to store in PostgreSQL or push to Elasticsearch. Passing format='json' forces the model to return only valid JSON without extraneous text.
import json
from ollama import chat
prompt = """
Extract the following fields: status, error_code, service_name from this log entry:
[2026-11-20 09:12:01] [payment-service] ERROR 504: Gateway Timeout when contacting bank gateway
"""
response = chat(
model='llama3.2:3b',
messages=[{'role': 'user', 'content': prompt}],
format='json'
)
# Parse directly into a Python dict without markdown ```json wrappers
data = json.loads(response['message']['content'])
print(f"Service: {data.get('service_name')} | Error code: {data.get('error_code')}")
4. Concurrent Processing with AsyncClient
Avoid using the synchronous client inside asynchronous web frameworks like FastAPI. An inference call taking 2-5 seconds will block the entire event loop, freezing the server. Instead, use AsyncClient alongside asyncio.gather.
import asyncio
from ollama import AsyncClient
async def analyze_task(client, task_id, prompt):
res = await client.generate(model='llama3.2:3b', prompt=prompt)
return task_id, res['response'][:60].strip()
async def main():
client = AsyncClient(host='http://localhost:11434')
tasks_data = [
("task_1", "Explain the concept of Kubernetes Ingress in one sentence"),
("task_2", "Explain the difference between Git merge and rebase"),
]
coros = [analyze_task(client, t_id, p) for t_id, p in tasks_data]
results = await asyncio.gather(*coros)
for t_id, summary in results:
print(f"[{t_id}] -> {summary}...")
if __name__ == '__main__':
asyncio.run(main())
Operational tip: To smoothly handle multiple requests in parallel, set the environment variable OLLAMA_NUM_PARALLEL=4 before starting the Ollama server.
Monitoring Resources and Real-World Performance
Running AI models on-premise requires closely monitoring VRAM and token generation speed. Without careful resource planning, GPU memory exhaustion will trigger Out of Memory (OOM) errors that crash processes.
1. Monitoring VRAM and GPU Offloading
To verify whether the model is fully loaded onto the GPU or overflowing into system RAM, check via your terminal:
# Monitor real-time NVIDIA VRAM usage
watch -n 1 nvidia-smi
# Inspect Ollama model load status
ollama ps
The PROCESSOR column in the ollama ps output should display 100% GPU. If you see a split such as 60% GPU / 40% CPU, token generation throughput will drop 4-5x due to PCIe bandwidth bottlenecks.
2. Benchmarking Inference Speed (Tokens Per Second)
The SDK returns detailed metadata after each request. You can extract these fields to export metrics to Prometheus or Grafana:
import ollama
res = ollama.generate(model='llama3.2:3b', prompt='Ping test')
# eval_duration is returned in nanoseconds
eval_count = res.get('eval_count', 0)
eval_duration_sec = res.get('eval_duration', 1) / 1e9
tokens_per_sec = eval_count / eval_duration_sec if eval_duration_sec > 0 else 0
print(f"Generated tokens: {eval_count}")
print(f"Inference speed: {tokens_per_sec:.2f} tokens/s")
Typically, a mid-range GPU such as an RTX 4060 (8GB VRAM) can achieve 45–65 tokens/s with llama3.2:3b. If you need to free VRAM immediately after finishing a batch script, add the keep_alive=0 parameter to your request so Ollama unloads the model from memory automatically.

