Don’t Let “Small Talk” Drain Your API Budget
Running GPT-4 or Claude 3.5 Sonnet is expensive. It’s painful to see your API bill at the end of the month just because of trivial questions. In reality, users often enter queries like “Hello”, “Who are you?”, or ask about football scores in a financial management app.
Each request like this consumes thousands of input tokens for system prompts and history. Instead of pushing everything to an expensive LLM, you can handle them statically or use a cheaper model. Semantic Router is the middleware layer that helps you automate this process.
This tool uses Vector Embeddings to analyze user intent in milliseconds. I applied this solution to a customer support chatbot with 10,000 requests/day. The results were surprising: API costs dropped by 45% and average response time fell from 2 seconds to just 0.5 seconds.
Environment Setup
You need Python 3.9 or higher to run this library. We will install the core version and the OpenAI driver for handling embeddings. If you want to maximize savings, you can replace OpenAI with local models later.
pip install -qU semantic-router openaiNext, configure your API Key. In a production environment, you should use a .env file for security.
export OPENAI_API_KEY="sk-xxx"Setting Up Smart Routes
The core of Semantic Router is Routes. Each Route represents a group of intents with associated sample sentences (utterances). The Router calculates semantic similarity to decide where the request should go.
Below is the configuration for 3 common groups: Chitchat, Technical Support, and Pricing.
from semantic_router import Route
# Chitchat category
chitchat = Route(
name="chitchat",
utterances=["hello", "hi admin", "who are you", "is anyone there", "good morning"],
)
# Technical issue handling
tech_support = Route(
name="technical",
utterances=["login error", "can't see the upload button", "forgot password", "API returned 500 error"],
)
# Conversion group
pricing = Route(
name="pricing",
utterances=["how much does it cost", "how to buy pro version", "are there any discounts", "annual maintenance fee"],
)To run this, we need a RouteLayer. This is the decision-making brain based on OpenAI’s text-embedding-3-small embedding model.
from semantic_router.layer import RouteLayer
from semantic_router.encoders import OpenAIEncoder
encoder = OpenAIEncoder()
rl = RouteLayer(encoder=encoder, routes=[chitchat, tech_support, pricing])Operation and Result Handling
When receiving a query from a user, RouteLayer returns the route name if it matches, or None if the question falls outside the defined groups. This approach helps you filter out noise before it reaches the primary LLM.
def handle_request(query):
guide = rl(query)
if guide.name == "chitchat":
return "Hello! I am your virtual assistant. How can I help you?"
if guide.name == "pricing":
return "You can check our pricing table at: itfromzero.com/pricing"
# If no route matches, then call GPT-4
return call_expensive_llm(query)3 Critical Lessons for Real-World Implementation
- Fine-tune Score Threshold: By default, the Router has its own confidence threshold. If the system misidentifies intents, adjust the
score_thresholdparameter in the RouteLayer to be more strict. - Prioritize Local Encoders: You should use
HuggingFaceEncoderwith theall-MiniLM-L6-v2model. It runs directly in RAM, allowing you to classify intents for free without depending on the internet. - Filter logs for optimization: Save queries that return
None. This is valuable data for adding to yourutterances, making the Router more accurate over time.
Semantic Router is more than just a money-saving tool. It is a protective layer that makes your AI application more professional and responsive. If you are building an AI Agent, integrate it into your first recognition layer immediately.
