Evaluation metrics

Overview

An evaluation metric is a number that summarises how good a model's predictions are. Accuracy, mean absolute error and area under the ROC curve are all metrics. Without them you cannot compare two models, tune a hyperparameter, decide whether a model is ready to ship or notice that it has started to degrade in production. With the wrong one, you can optimise confidently towards the wrong goal.

There is no single best metric. Each answers a different question. Does the model rank cases in the right order? Are its probabilities trustworthy? How large are its errors in the units the business cares about? How often does it miss the rare case that really matters? A good evaluation picks one primary metric that reflects the real cost of mistakes, and a few secondary metrics that catch problems the primary one hides.

An everyday analogy: judging a weather forecaster. You could count how often they say "rain" on rainy days, but someone who says "no rain" every day in a desert would score very well and tell you nothing. You might instead check whether "70 per cent chance of rain" really means rain on about 70 per cent of such days, or how badly they miss on the days that matter for your picnic. Each of these is a different metric, and each rewards a different kind of forecaster.

What it is

Two distinctions organise the whole field. The first is between threshold metrics, which judge hard decisions, and threshold-free metrics, which judge scores or probabilities across all possible thresholds. The second is between proper scoring rules, which are optimised in expectation only by reporting your true belief, and other metrics, which can be gamed by distorting probabilities.

The metric you optimise during training (the loss) and the metric you report need not be the same. A model may be trained with cross-entropy and judged on recall at a fixed precision. Keeping the two aligned, at least in spirit, is part of good model design.

Where it came from

Precision and recall come from information retrieval, where researchers in the 1950s and 1960s measured how many retrieved documents were relevant and how many relevant documents were retrieved. C. J. van Rijsbergen's 1979 book "Information Retrieval" defined an effectiveness measure that combines the two, the origin of what is now the F-measure.

The receiver operating characteristic curve comes from signal detection theory, developed for radar during the Second World War, and later spread to psychology and medicine. James Hanley and Barbara McNeil's 1982 paper in Radiology, "The meaning and use of the area under a receiver operating characteristic (ROC) curve", showed that the area equals the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative one. Takaya Saito and Marc Rehmsmeier argued in PLOS ONE in 2015 that precision-recall plots are more informative than ROC plots on imbalanced data.

Glenn Brier proposed his score for probability forecasts in 1950 in Monthly Weather Review. The continuous ranked probability score was developed in the 1970s, including work by James Matheson and Robert Winkler in 1976, and Tilmann Gneiting and Adrian Raftery's 2007 paper "Strictly proper scoring rules, prediction, and estimation" in the Journal of the American Statistical Association is the standard modern reference on proper scoring. Mean reciprocal rank was used in the TREC-8 question-answering track in 1999. Kalervo Järvelin and Jaana Kekäläinen formalised cumulated gain measures, including NDCG, in ACM Transactions on Information Systems in 2002. Chuan Guo and colleagues' "On calibration of modern neural networks" (ICML 2017) showed that deep networks are often over-confident and popularised temperature scaling.

How it works

The confusion matrix

For a binary classifier at a given threshold, every prediction falls into one of four cells: true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Almost every classification metric is a ratio of these cells.

Accuracy   = (TP + TN) / (TP + FP + TN + FN)
Precision  = TP / (TP + FP)
Recall     = TP / (TP + FN)
F1         = 2 · Precision · Recall / (Precision + Recall)
Log loss   = −(1/N) · Σ [ yᵢ · log pᵢ + (1 − yᵢ) · log(1 − pᵢ) ]
Brier      = (1/N) · Σ (pᵢ − yᵢ)²
MAE        = (1/N) · Σ |yᵢ − ŷᵢ|
RMSE       = √( (1/N) · Σ (yᵢ − ŷᵢ)² )
R²         = 1 − Σ (yᵢ − ŷᵢ)² / Σ (yᵢ − ȳ)²
DCG@k      = Σ_{r=1..k} relᵣ / log₂(r + 1),   NDCG@k = DCG@k / ideal DCG@k

TP, FP, TN and FN are the counts of true positives, false positives, true negatives and false negatives; N is the number of examples; yᵢ is the true label (0 or 1 for classification, a number for regression); pᵢ is the predicted probability of the positive class; ŷᵢ is a predicted value; ȳ is the mean of the true values; relᵣ is the relevance grade of the item at rank r; and the ideal DCG is the DCG of the best possible ordering. In the version shown, DCG uses the graded relevance directly; some libraries use 2^rel − 1 instead.

Threshold metrics

Accuracy is the share of correct predictions. It is easy to explain and misleading on imbalanced data: if 1 per cent of transactions are fraud, a model that never flags fraud is 99 per cent accurate. Precision asks: of the cases flagged, how many were right? Recall (sensitivity, true positive rate) asks: of the real positives, how many were found? Raising the threshold usually increases precision and lowers recall. F1 is their harmonic mean, which is low if either is low; F-beta weights recall β times as much as precision. Specificity is the true negative rate, and balanced accuracy is the average of recall and specificity. For multi-class problems, macro-averaging treats every class equally, while micro-averaging pools all decisions and is dominated by frequent classes.

Threshold-free metrics

The ROC curve plots true positive rate against false positive rate as the threshold sweeps from high to low. ROC-AUC is the area under it: 0.5 is random ranking and 1.0 is perfect. Because the false positive rate divides by the large number of negatives, ROC-AUC can look excellent on imbalanced data even when most flagged cases are false alarms. The precision-recall curve plots precision against recall, and its summary, average precision (PR-AUC), has a baseline equal to the positive rate rather than 0.5. On rare-event problems it is usually the more honest view.

Probabilistic metrics and calibration

Log loss (cross-entropy) heavily punishes confident wrong answers; predicting probability 0.01 for something that happens costs far more than predicting 0.4. The Brier score is the mean squared difference between probabilities and outcomes; it is bounded between 0 and 1 and less harsh on extremes. Both are proper scoring rules. Calibration asks whether predicted probabilities match observed frequencies. A reliability diagram groups predictions into bins and plots average predicted probability against the observed positive rate; a perfectly calibrated model lies on the diagonal. Expected calibration error summarises the gaps. A model can rank perfectly (high AUC) and still be badly calibrated, which matters whenever probabilities feed a cost calculation. Recalibration methods include Platt scaling, isotonic regression and temperature scaling; scikit-learn's CalibratedClassifierCV gained a method="temperature" option in version 1.8.

Regression metrics

MAE is the average absolute error in the target's units and is robust to a few large errors. RMSE squares errors before averaging, so it punishes large errors more and is always at least as large as MAE; a big gap between them signals occasional large misses. MAPE expresses error as a percentage of the true value, which is easy to communicate but explodes when true values are near zero and penalises over-forecasts and under-forecasts differently. R² is the share of variance explained relative to predicting the mean; it can be negative on held-out data when a model is worse than that constant baseline. Pick MAE when all errors cost about the same per unit, RMSE when large errors are disproportionately costly.

Ranking metrics

For search, recommendation and retrieval, what matters is the order. Precision@k and recall@k count relevant items in the top k. MRR averages 1 divided by the rank of the first relevant result, which suits tasks with one right answer. NDCG@k gives graded credit for relevant items and discounts it logarithmically by position, then normalises by the best possible ordering so that scores lie between 0 and 1. These are the standard metrics in learning to rank and retrieval systems such as BM25 pipelines.

Probabilistic forecast metrics

When a model outputs a whole distribution, as in many forecasting models such as Chronos, point metrics are not enough. The CRPS compares the predicted cumulative distribution with the observed value and reduces to MAE when the forecast is a single point. The pinball (quantile) loss scores individual quantiles. Coverage checks intervals: a 90 per cent interval should contain the truth about 90 per cent of the time. Report coverage together with interval width, because very wide intervals cover everything and help no one.

Choosing metrics for imbalanced data

1. Report the positive rate, so readers know the baseline.
2. Use PR-AUC or recall at a fixed precision (or precision at a fixed recall) rather than accuracy or ROC-AUC alone.
3. Choose the operating threshold from costs: estimate what a false positive and a false negative each cost, and pick the threshold that minimises expected cost on validation data.
4. Use stratified cross-validation so each fold contains enough positives.
5. Check calibration if probabilities will be used directly; resampling and class weights distort probabilities.

A scikit-learn example

from sklearn.metrics import (average_precision_score, brier_score_loss,
    classification_report, confusion_matrix, log_loss, roc_auc_score)

proba = model.predict_proba(X_test)[:, 1]
pred = (proba >= 0.3).astype(int)            # threshold chosen on validation data

print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))
print("ROC-AUC", roc_auc_score(y_test, proba))
print("PR-AUC ", average_precision_score(y_test, proba))
print("Log loss", log_loss(y_test, proba))
print("Brier  ", brier_score_loss(y_test, proba))

For regression, mean_absolute_error, root_mean_squared_error, mean_absolute_percentage_error and r2_score live in the same module, and ndcg_score covers ranking.

Key concepts and terms

A worked example

This is an illustrative scenario. A clinic screens 10,000 patients for a condition that affects 2 per cent, so 200 are positive. A model outputs a risk score.

At threshold 0.5: the model flags 150 patients, of whom 120 are positive. So TP = 120, FP = 30, FN = 80, TN = 9,770. Accuracy is (120 + 9,770) / 10,000 = 98.9 per cent, barely above the 98 per cent of a model that flags nobody. Precision is 120 / 150 = 0.80. Recall is 120 / 200 = 0.60. F1 is 2 × 0.80 × 0.60 / 1.40 ≈ 0.69.

At threshold 0.2: the model flags 400 patients, of whom 176 are positive. Precision falls to 0.44, recall rises to 0.88, F1 ≈ 0.59. Accuracy falls to 97.5 per cent.

Which is better? The clinic says a missed case costs about ten times a needless follow-up test. Expected cost at 0.5 is 80 × 10 + 30 × 1 = 830 units; at 0.2 it is 24 × 10 + 224 × 1 = 464 units. The lower threshold wins, even though F1 and accuracy both prefer the higher one.

Threshold-free view: ROC-AUC is 0.95, which sounds excellent, while PR-AUC is 0.61 against a baseline of 0.02. Both are true; the PR view shows more clearly how many false alarms come with each catch. A reliability diagram shows the model is over-confident above 0.6, so the team applies isotonic calibration on a validation split before using the scores in a risk letter.

Strengths and limitations

Strengths:

• Make model comparison objective and repeatable.
• Different metrics expose different failure modes: ranking, calibration, error size, rare-class misses.
• Proper scoring rules give honest incentives for probability estimates.
• Standard implementations in libraries such as scikit-learn make them easy to compute consistently.

Limitations:

• Every metric hides something; a single number never tells the whole story.
• Threshold metrics depend on a threshold that is often chosen arbitrarily.
• Metrics computed on a small test set have wide uncertainty that is rarely reported.
• Optimising hard for one metric can degrade others, such as calibration, that matter later.
• Offline metrics may not match online business outcomes, which need A/B tests or evals in context.

How it compares with alternatives

Practical guidance and best practices

1. Write down what a false positive and a false negative cost before choosing a metric.
2. Pick one primary metric for decisions and two or three secondary ones as guards.
3. Always compare against a trivial baseline, such as predicting the majority class or the mean.
4. Report uncertainty, for example with cross-validation spread or bootstrap confidence intervals.
5. On imbalanced data, lead with PR-AUC or recall at fixed precision, and state the positive rate.
6. Choose thresholds on validation data, never on the test set.
7. Check calibration with a reliability diagram whenever probabilities will be used as probabilities.
8. Slice metrics by important subgroups, such as region or customer type, to catch hidden failures.
9. Keep the evaluation code and test set fixed across experiments so numbers stay comparable.

Common mistakes

• Reporting accuracy on a heavily imbalanced dataset.
• Treating a high ROC-AUC as proof that precision will be acceptable in production.
• Using MAPE when some true values are zero or near zero.
• Comparing RMSE across datasets with different units or scales.
• Tuning the threshold on the test set and then reporting test metrics as if they were unbiased.
• Averaging F1 across classes without saying whether it is macro, micro or weighted.
• Forgetting that resampling or class weights distort predicted probabilities.

Frequently asked questions

When should I use ROC-AUC and when PR-AUC?

Use ROC-AUC when classes are reasonably balanced or when you care about ranking across the whole range. Use PR-AUC when positives are rare and you mainly care about the top of the ranking, because it reflects how many false alarms accompany each true detection. Reporting both, together with the positive rate, is common and harmless.

What is a good F1 score?

There is no universal answer. F1 depends heavily on the positive rate and on the difficulty of the task. Compare against a baseline, such as flagging everything or a simple model, and against the costs of each error type. F1 also ignores true negatives and treats precision and recall as equally important, which may not match your situation.

Why is my R² negative?

On held-out data, R² is negative when the model's squared error is larger than that of simply predicting the mean of the true values. It usually means the model is badly overfitted, has a bug such as misaligned rows, or faces a distribution shift. On training data with an intercept, ordinary least squares cannot produce a negative R².

Is log loss or Brier score better?

Both are proper scoring rules. Log loss punishes confident mistakes far more severely and connects directly to likelihood and information theory, so it suits training and model comparison. The Brier score is bounded, easier to explain and less dominated by a few extreme errors, so it is popular for reporting forecast quality. Many teams report both.

How do I choose a classification threshold?

Estimate the cost of each error type and the benefit of each correct decision, then pick the threshold that maximises expected value on validation data. If costs are unknown, pick a threshold that meets an operational constraint, such as the number of cases a team can review per day, or a minimum precision. The default of 0.5 is rarely optimal for imbalanced problems.

What metrics suit ranking and recommendation?

Use NDCG@k when relevance is graded, MRR when there is one correct answer per query, and precision@k or recall@k when relevance is binary and the list length is fixed. Compute them per query or per user and then average, and check that the evaluation set includes realistic candidates, not only the items users already clicked.

What does coverage tell me about prediction intervals?

Coverage is the share of true values that fall inside the predicted intervals. A well-behaved 90 per cent interval should cover about 90 per cent of cases. Coverage alone is not enough, because extremely wide intervals always cover; report average width too, or use a score such as CRPS or interval score that balances both.

Glossary

• True positive: a positive case correctly predicted as positive.
• False negative: a positive case wrongly predicted as negative.
• Threshold: the score above which a case is labelled positive.
• Specificity: the share of actual negatives correctly predicted as negative.
• Balanced accuracy: the average of recall and specificity.
• Average precision: the mean precision over all recall levels, a common estimate of PR-AUC.
• Reliability diagram: a plot of predicted probability against observed frequency.
• Temperature scaling: dividing logits by a single fitted constant to improve calibration.
• MRR: the mean of 1 divided by the rank of the first relevant result.
• Pinball loss: the loss used to score quantile forecasts.

Summary

Evaluation metrics translate predictions into numbers that guide every decision in a machine learning project. Classification uses the confusion matrix and its ratios (accuracy, precision, recall, F1), threshold-free ranking measures (ROC-AUC, PR-AUC) and probability scores (log loss, Brier score) with calibration checks. Regression uses MAE, RMSE, MAPE and R², each with its own sensitivity to large errors and scale. Ranking uses precision@k, MRR and NDCG, and probabilistic forecasts use CRPS, pinball loss and interval coverage. Choose metrics from the real costs of mistakes, lead with PR-based measures on imbalanced data, set thresholds on validation data, report uncertainty and baselines, and never trust a single number on its own.

Related

It belongs to Training fundamentals in my stack, under Machine learning. Others in the same group: