Özer Cem KELAHMETKELAHMET
Topics · MLOps and AI infrastructure

Monitoring and evaluating models in production: drift, tracing and evals

A deployed model degrades silently as the world changes. This page covers what to monitor, how to evaluate LLM applications continuously, and how to close the loop back to training.

Overview

Software fails loudly; models fail quietly. Inputs drift, upstream tables change, user behaviour shifts and predictions slowly get worse without any error message. Monitoring watches the inputs, the outputs and, when labels arrive, the accuracy.

What to monitor

Signal Detects Tools
Input distributions Data drift, broken pipelines Model monitoring, schema drift detection
Prediction distributions Behaviour changes Prometheus, Grafana
Accuracy on delayed labels Real performance Evaluation jobs
Latency, errors, cost Service health OpenTelemetry
LLM traces Wrong tools, bad retrieval, loops Tracing, Langfuse, LangSmith, Phoenix

Drift, in practice

Compare live data with the training data feature by feature using population stability or distance measures, and alert on large shifts in important features. Not every drift hurts; confirm with accuracy when labels arrive, and retrain on a schedule or when performance drops.

Evaluating LLM applications continuously

Keep an offline evaluation set and run it on every change. In production, sample traces and score them with rules, LLM-as-judge checks and user feedback; tools include DeepEval, RAGAS, TruLens, Braintrust and promptfoo. Add every confirmed failure to the evaluation set.

Closing the loop

Monitoring feeds three actions: alerts for incidents, new evaluation cases for regressions, and new training data for the next model. Active learning chooses which production cases are worth labelling.

Frequently asked questions

How often should a model be retrained?

When monitoring shows a drop, or on a schedule that matches how fast the data changes, from daily for fraud to yearly for slow processes.

What is the minimum monitoring for an LLM app?

Traces for every request, cost and latency per feature, and a weekly review of sampled traces against the evaluation set.

Summary

Watch inputs, outputs and accuracy; trace every LLM request; evaluate continuously; and turn every failure into a test and better training data.

Back to the topic

This page is part of MLOps and AI infrastructure, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open