# Monitoring and evaluating models in production: drift, tracing and evals

A deployed model degrades silently as the world changes. This page covers what to monitor, how to evaluate LLM applications continuously, and how to close the loop back to training.

## Overview

Software fails loudly; models fail quietly. Inputs drift, upstream tables change, user behaviour shifts and predictions slowly get worse without any error message. Monitoring watches the inputs, the outputs and, when labels arrive, the accuracy.

## What to monitor

| Signal | Detects | Tools |
|---|---|---|
| Input distributions | Data drift, broken pipelines | [Model monitoring](/tools/model-monitoring), [schema drift detection](/tools/schema-drift-detection) |
| Prediction distributions | Behaviour changes | [Prometheus](/tools/prometheus), [Grafana](/tools/grafana) |
| Accuracy on delayed labels | Real performance | Evaluation jobs |
| Latency, errors, cost | Service health | [OpenTelemetry](/tools/opentelemetry) |
| LLM traces | Wrong tools, bad retrieval, loops | [Tracing](/tools/tracing), [Langfuse](/tools/langfuse), [LangSmith](/tools/langsmith), [Phoenix](/tools/phoenix) |

## Drift, in practice

Compare live data with the training data feature by feature using population stability or distance measures, and alert on large shifts in important features. Not every drift hurts; confirm with accuracy when labels arrive, and retrain on a schedule or when performance drops.

## Evaluating LLM applications continuously

Keep an offline evaluation set and run it on every change. In production, sample traces and score them with rules, [LLM-as-judge](/tools/llm-as-judge) checks and user feedback; tools include [DeepEval](/tools/deepeval), [RAGAS](/tools/ragas), [TruLens](/tools/trulens), [Braintrust](/tools/braintrust) and [promptfoo](/tools/promptfoo). Add every confirmed failure to the evaluation set.

## Closing the loop

Monitoring feeds three actions: alerts for incidents, new evaluation cases for regressions, and new training data for the next model. [Active learning](/tools/active-learning) chooses which production cases are worth labelling.

## Frequently asked questions

### How often should a model be retrained?

When monitoring shows a drop, or on a schedule that matches how fast the data changes, from daily for fraud to yearly for slow processes.

### What is the minimum monitoring for an LLM app?

Traces for every request, cost and latency per feature, and a weekly review of sampled traces against the evaluation set.

## Summary

Watch inputs, outputs and accuracy; trace every LLM request; evaluate continuously; and turn every failure into a test and better training data.

## Back to the topic

This page is part of **[MLOps and AI infrastructure](/topics/mlops-ai-infrastructure)**, which maps every concept, tool, simulation and article on the subject.
