Posted inAI

Shedding Light on the ‘Black Box’ of Local RAG with Arize Phoenix in 5 Minutes

Artificial Intelligence tutorial - IT technology blog
Artificial Intelligence tutorial - IT technology blog

Quick Start: Run Arize Phoenix with Just 2 Commands

If you already have a Python environment set up, it takes less than a minute to launch Arize Phoenix. Instead of struggling with lengthy YAML configuration files, let’s dive straight into execution.

First, install the necessary libraries via pip:

pip install arize-phoenix openinference-instrumentation-llama-index

Next, open your terminal or a Python file and run this code to activate the monitoring server:

import phoenix as px

# Launch Phoenix server
session = px.launch_app()

# Print the dashboard access URL
print(f"Dashboard is running at: {session.url}")

The web interface will immediately appear at http://localhost:6006. This is where the entire “inner workings” of your RAG system will be exposed for close monitoring.

Why I Chose Arize Phoenix over LangSmith?

When I first started with RAG, I often struggled with incorrect model answers without knowing the root cause. Was the Vector Database retrieving the wrong documents? Was the Model hallucinating despite having the correct context? Or was my Prompt simply terrible?

LangSmith is a great choice, but it runs on the Cloud. For enterprise projects with strict security requirements, sending internal data to a third-party server is an unacceptable risk. Arize Phoenix solves this problem completely.

The advantages that keep me using this tool:

  • Absolute Privacy: Data stays on your machine; not a single byte leaves.
  • OpenTelemetry Standard: You aren’t locked into any specific framework.

    Whether you use LangChain, LlamaIndex, or raw code, Phoenix can handle it all.

  • Deep Tracing: You can see exactly how many milliseconds the Embedding step took, or which specific text chunks were retrieved.
  • Zero Cost: The open-source version provides more than enough power for personal needs and startups.

Installing “CCTV” into Your RAG Code

To allow Phoenix to record everything, we need to use Instrumentation. Think of this as installing dashcams at every junction of your data processing flow.

Here is the 100% local RAG stack I usually use: LlamaIndex + Ollama + Arize Phoenix.

import phoenix as px
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, set_global_handler

# 1. Activate Phoenix
px.launch_app()

# 2. Connect LlamaIndex to Phoenix
set_global_handler("arize_phoenix")

# 3. Basic RAG flow
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

# Test a query
response = query_engine.query("How do I install Arize Phoenix?")
print(response)

Once finished, return to the dashboard. A new “Trace” entry will appear. Click on it, and you’ll see a tree diagram showing exactly: the Embedding step took 200ms, Retrieval pulled 3 text chunks, and the LLM took 2 seconds to synthesize the answer.

Using an LLM as an “Automated Proctor” for Scoring

The standout feature of Phoenix is its Evaluation capability. Manually reading hundreds of logs to check accuracy is extremely time-consuming. Instead, Phoenix allows you to use another LLM (an LLM Judge) to score your main LLM.

You can leverage Ollama to run Llama3 or Mistral as the judge, saving you OpenAI API costs:

from phoenix.evals import HallucinationEvaluator, OpenAIModel
# Configure the scoring model (pointing to Ollama's local endpoint)
judge_model = OpenAIModel(base_url="http://localhost:11434/v1", api_key="ollama")

evaluator = HallucinationEvaluator(model=judge_model)
# Phoenix will automatically label answers as "Pass" or "Fail" based on the context.

This feature is incredibly useful when you change prompts. Just run your test suite and look at the dashboard to see if accuracy increased or decreased—no more guessing.

Real-World Tips to Avoid Headaches

After months of deploying RAG, I’ve gathered a few key tips to save you time:

  1. Manage RAM: Phoenix stores traces directly in the cache. If you run thousands of requests continuously, your machine could consume 8GB-16GB of RAM and freeze. Remember to clear the dataset periodically.
  2. Library Conflicts: Phoenix updates its features weekly. If you encounter strange errors, create a new venv and update both arize-phoenix and llama-index to the latest versions simultaneously.
  3. Don’t Trust the AI Judge 100%: While LLM Judges are convenient, they can still make mistakes (false positives). You should spend time spot-checking about 5-10% of the results to ensure reliability.
  4. Optimize Latency: Use Phoenix to pinpoint which step is slowing down the system. Sometimes, simply switching the Embedding model can reduce latency from 2 seconds to 200ms.

Monitoring isn’t just for fixing bugs; it’s for understanding how your data “talks” to the model. With Arize Phoenix, you have a powerful microscope to optimize your RAG system while keeping your data secrets safe.

Share:
Tags: