Comparing 3 Approaches to Integrating Multiple LLMs in Python
If you’ve ever called OpenAI, Claude, and Gemini simultaneously in the same project, you know the frustration of handling divergent payload formats. OpenAI expects messages as a list of dicts. Anthropic isolates the system parameter. Meanwhile, Google Gemini requires wrapping payload data inside contents and parts.
When tackling multi-model integration, engineering teams typically choose one of three approaches:
- Approach 1: Use Official SDKs Directly (openai, anthropic, google-genai). Build custom wrappers to map inputs and outputs across vendors.
- Approach 2: Use High-Level Frameworks like LangChain or LlamaIndex. Rely on built-in abstraction layers to switch providers by swapping class loaders.
- Approach 3: Use LiteLLM (Python Library / Proxy). Retain 100% of the standard OpenAI Chat Completion format, changing only the model name (e.g.,
claude-3-5-sonnet-20240620orgemini/gemini-1.5-flash).
Evaluating the Pros and Cons of Each Solution
1. Building Custom Wrappers Around Official SDKs
This gives your team granular control over every byte sent and received, avoiding external runtime dependencies. However, maintenance costs are notoriously steep.
Every time a provider launches a new model or alters its token usage response schema, your wrapper risks breaking. Normalizing streaming and async patterns across three different SDKs is also a frequent source of subtle bugs.
2. LangChain or LlamaIndex
Both frameworks excel when engineering complex autonomous agents, multi-step RAG pipelines, or orchestration graphs. However, if all you need is straightforward API calling, LangChain introduces unnecessary overhead and bloat.
Imagine your production system running into 429 rate-limit bottlenecks. Tracing call stacks through dozens of deeply nested LangChain abstraction layers can easily turn troubleshooting into a multi-hour ordeal.
3. The LiteLLM Library
LiteLLM tackles a single core challenge: translating API syntax from various providers (Anthropic, Gemini, Mistral, Cohere, Bedrock, Ollama) into OpenAI’s standard completion() signature. The return value is consistently a standard response object with familiar fields like choices[0].message.content and usage.total_tokens.
You just need to include the vendor prefix when required (such as gemini/ or bedrock/). The library ships new features on a weekly basis, so pinning a specific version in your requirements.txt is recommended to avoid regressions.
When Should Your Project Switch to LiteLLM?
Suppose your backend runs smoothly on the OpenAI SDK. Suddenly, requirements change: ‘Add automatic failover to Claude when OpenAI is throttled’ or ‘Offload classification workloads to Gemini 1.5 Flash to slash 70% of API costs’. This is precisely where LiteLLM shines.
Instead of rewriting prompt parsers or JSON parsing logic, you modify a single import statement and leave the rest of your data processing pipeline untouched.
Step-by-Step Guide to Implementing LiteLLM
Step 1: Install the Library and Configure the Environment
Install the core LiteLLM package along with python-dotenv via pip:
pip install litellm python-dotenvCreate a .env file in your project root to store your API keys:
OPENAI_API_KEY="sk-..."
ANTHROPIC_API_KEY="sk-ant-..."
GEMINI_API_KEY="AIza..."Step 2: Invoke Multiple Models via a Single Function
The code below demonstrates calling GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Flash without changing a single line of response parsing logic:
import os
from dotenv import load_dotenv
from litellm import completion
load_dotenv()
messages = [
{"role": "system", "content": "You are a concise technical IT assistant."},
{"role": "user", "content": "Briefly explain what a Docker container is in 2 sentences."}
]
# 1. Call GPT-4o
res_openai = completion(
model="gpt-4o",
messages=messages
)
print("OpenAI:", res_openai.choices[0].message.content)
# 2. Call Claude 3.5 Sonnet (LiteLLM translates to Anthropic format automatically)
res_claude = completion(
model="claude-3-5-sonnet-20240620",
messages=messages
)
print("Claude:", res_claude.choices[0].message.content)
# 3. Call Google Gemini (Add gemini/ prefix)
res_gemini = completion(
model="gemini/gemini-1.5-flash",
messages=messages
)
print("Gemini:", res_gemini.choices[0].message.content)Step 3: Automated Fallbacks for High Availability
A production-critical feature is the fallbacks parameter. When your primary model hits a rate limit (429) or vendor servers encounter downtime (500/503), LiteLLM automatically routes the request to secondary models without manual intervention:
from litellm import completion
try:
response = completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Extract structured JSON from the following text..."}],
fallbacks=["claude-3-5-sonnet-20240620", "gemini/gemini-1.5-flash"],
timeout=10 # Automatically try the next fallback if no response within 10s
)
print(response.choices[0].message.content)
except Exception as e:
print(f"All fallback models failed: {e}")Step 4: Real-Time Stream Processing
When developing chat interfaces, streaming is essential for optimizing time-to-first-token (TTFT). Consuming stream chunks with LiteLLM uses identical syntax to the official OpenAI client:
response = completion(
model="claude-3-5-sonnet-20240620",
messages=[{"role": "user", "content": "Write a bash script to back up a MySQL database."}] ,
stream=True
)
for chunk in response:
content = chunk.choices[0].delta.content or ""
print(content, end="", flush=True)Tips for Parsing Structured Output from LLMs
Many models tend to prepend conversational greetings or enclose data inside markdown “`json blocks. Developers frequently turn to regular expressions to extract clean JSON payloads.
If you need to quickly validate regex patterns without constantly rerunning code, the regex tester tool at toolcraft.app/en/tools/developer/regex-tester lets you test expressions and view match highlights directly in your browser.
Once your regex is verified, integrate it into your Python parsing logic to safely extract JSON without syntax errors.
Real-Time Cost Tracking per Request
Rather than waiting until the end of the billing cycle to inspect invoices across OpenAI, Anthropic, and Google Cloud consoles, LiteLLM calculates cost metrics directly on every individual completion:
from litellm import completion, completion_cost
resp = completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Write a poem about IT."}]
)
cost = completion_cost(completion_response=resp)
print(f"Input tokens: {resp.usage.prompt_tokens}")
print(f"Output tokens: {resp.usage.completion_tokens}")
print(f"Cost for this request: ${cost:.6f} USD")Cost metrics can be written straight to your database or application logs, making it simple to enforce granular budget tracking per user or per feature.
