Phoenix (Arize)

Overview

Phoenix is an open-source observability and evaluation tool from Arize AI for applications built on large language models (LLMs), including retrieval-augmented generation (RAG) systems and agents. It collects traces of what an application does, lets you score those traces with automated or human evaluations, and supports datasets, experiments and prompt versioning. It can run on a laptop, in a notebook or as a self-hosted server.

Phoenix is built on OpenTelemetry, the open standard for traces and metrics. Its companion project, OpenInference, defines how LLM-specific details such as prompts, retrieved documents, tool calls and token counts are recorded in OpenTelemetry spans. Because of that, the same instrumentation can send data to Phoenix or to any other OpenTelemetry-compatible backend.

An everyday analogy: a home security camera system that uses standard video formats. You can watch the footage on the system's own screen, but because the format is standard, you can also play it on any other player. Phoenix is the screen; OpenInference and OpenTelemetry are the standard format.

What it is

Phoenix is released under the Elastic License 2.0. Its documentation states that self-hosting on your own infrastructure is free and that features are not gated. The Elastic License is source-available rather than an OSI-approved open-source licence, and it restricts offering the software as a managed service to third parties, so read it if you plan to resell hosting.

Where it came from

Arize AI was founded in 2020 by Jason Lopatecki and Aparna Dhinakaran to build monitoring for machine learning models in production. As LLM applications spread, the company released Phoenix in 2023 as an open-source tool for troubleshooting LLMs and embeddings, first aimed at notebooks.

The project then moved toward open standards. Arize created OpenInference to describe LLM operations in OpenTelemetry terms and published instrumentation packages for many model providers and frameworks. Phoenix became a full OpenTelemetry collector with a persistent server, projects, sessions, datasets, experiments, a prompt playground and prompt management. Its major versions have changed client APIs several times; the current Python client lives in a separate arize-phoenix-client package, and the evals library in arize-phoenix-evals.

Arize also sells a commercial platform, Arize AX, for larger enterprise deployments. Phoenix remains the self-hostable, source-available product.

How it works

1. Start a Phoenix server

You can run Phoenix as a container, install it with pip and start it locally, or launch it inside a Jupyter notebook. A single container stores data in SQLite. For durable, multi-user use you point it at PostgreSQL with PHOENIX_SQL_DATABASE_URL.

2. Instrument the application

You register a tracer that exports to Phoenix and enable OpenInference instrumentors.

from phoenix.otel import register

tracer_provider = register(
    project_name="support-bot",
    endpoint="http://localhost:6006/v1/traces",
    auto_instrument=True,  # enable installed OpenInference instrumentors
)

With auto_instrument=True, every installed OpenInference package hooks into its library. If two libraries would both record the same model call, enable only one instrumentor to avoid duplicate spans. Custom functions can be wrapped with OpenTelemetry spans.

3. Read traces

Each request appears as a trace of spans with kinds such as LLM, retriever, tool, chain, embedding and agent. A retriever span lists the documents returned with their scores. An LLM span shows the full prompt, response, model and tokens. Sessions group the traces of one conversation.

4. Evaluate

The evals library runs judges over spans or dataframes. For a RAG app, typical checks are document relevance for each retrieved chunk, faithfulness of the answer to the context and correctness against a reference. Results are logged back as annotations, so you can filter the UI for unfaithful answers or irrelevant retrievals. See LLM as judge.

5. Experiment

You build a dataset from examples or from selected spans, then run an experiment: a task function over every example, scored by evaluators. Experiments are stored and compared in the UI. Prompts can be versioned, tagged and replayed in the playground against stored spans.

Key concepts and terms

A worked example

This is an illustrative scenario.

A small team builds a RAG assistant over 1,200 internal engineering documents using LlamaIndex. Users say answers are sometimes off-topic.

1. They run Phoenix in Docker, install the LlamaIndex OpenInference instrumentor and call register with auto_instrument=True.
2. After one day, Phoenix holds 900 traces. Opening a few shows retriever spans returning five chunks each.
3. They run a document relevance judge on all retrieved chunks. About 31% are marked irrelevant, concentrated in questions that mention product code names.
4. They also run a faithfulness judge on answers. Unfaithful answers mostly coincide with traces where four or five chunks were irrelevant.
5. They create a dataset of 80 such questions and run two experiments: the current retriever, and one that adds keyword matching through hybrid search and a reranker. Irrelevant chunks drop to 12% and faithfulness rises.
6. They review 30 judge labels by hand and agree with 26, which gives them reasonable confidence in the trend.

Strengths and limitations

Strengths:

• Built on OpenTelemetry, so instrumentation is portable.
• Wide auto-instrumentation coverage through OpenInference.
• Strong RAG tooling: retriever spans, document relevance and faithfulness evaluations.
• Runs anywhere, from a notebook to a self-hosted server, with no feature gating.
• Datasets, experiments and prompt management in the same tool.

Limitations:

• Elastic License 2.0 is source-available, not OSI open source, and limits offering it as a hosted service.
• SQLite storage suits trials; production needs PostgreSQL and operational care.
• Frequent API changes between major versions can break older code.
• LLM-judge evaluations add model cost and must be checked against humans.
• Operating it at scale is your responsibility when self-hosting.

How it compares with alternatives

Practical guidance and best practices

1. Start locally to learn the trace views, then move to a server with PostgreSQL.
2. Use one project per application and environment.
3. Enable only the instrumentors you need to avoid duplicate spans.
4. Add session and user ids to spans so conversations can be followed.
5. Evaluate retrieval separately from generation; many RAG failures start in retrieval.
6. Turn failing traces into dataset examples and test fixes with experiments.
7. Check a sample of judge labels by hand before acting on them.
8. Pin Phoenix and client versions, and read release notes before upgrading.

Common mistakes

• Running production on default SQLite storage without backups.
• Double instrumenting a call through two frameworks.
• Looking only at final answers and ignoring retriever spans.
• Treating judge scores as exact rather than as estimates.
• Upgrading across major versions without checking client API changes.

Frequently asked questions

Is Phoenix free?

Self-hosting Phoenix is free under the Elastic License 2.0, and the documentation states there is no feature gating. Arize also sells a separate commercial platform for enterprises.

Do I have to use Phoenix's own SDK?

No. Phoenix accepts standard OpenTelemetry traces. The phoenix.otel helper and OpenInference packages make setup easier, but any OTLP exporter that follows the conventions works.

What is OpenInference?

It is a set of semantic conventions and instrumentation packages, created by Arize, that describe LLM operations such as prompts, retrieved documents, tool calls and token counts inside OpenTelemetry spans.

Can Phoenix evaluate RAG systems?

Yes. It records retrieved documents on retriever spans and includes evaluators for document relevance, faithfulness and correctness, whose results can be attached to traces.

Where does Phoenix store data?

A default deployment uses SQLite in a working directory. Production deployments usually configure PostgreSQL through an environment variable.

Glossary

• Observability: understanding system behaviour from its recorded outputs.
• RAG: retrieval-augmented generation, where a model answers using retrieved documents.
• Semantic conventions: agreed names for attributes recorded in telemetry.
• Collector: a service that receives telemetry data.
• Source-available: code you can read and run, under terms that are not OSI open source.
• Faithfulness: whether an answer is supported by its context.
• Instrumentor: a package that adds tracing to a library automatically.

Summary

Phoenix is Arize's self-hostable tool for tracing and evaluating LLM, RAG and agent applications, built on OpenTelemetry with OpenInference conventions. It is especially useful for RAG debugging, because retrieval and generation can be traced and judged separately. It runs anywhere from a notebook to a PostgreSQL-backed server under the Elastic License 2.0. Use open instrumentation, evaluate retrieval and answers separately, verify judges by hand and manage upgrades carefully. See tracing and evals for the broader practice.

Related

It belongs to LLM observability and evaluation in my stack, under Generative AI and LLMs. Others in the same group: