A score is only as good as its report
"96% accurate" means almost nothing on its own. It depends on what was held out, what the baseline was, and where the model is weakest.
What a held-out report states
How well a system predicts rows whose target values it never saw. A good report records the split, the seed, the baseline, and the exact data snapshot behind the score. A common default is a stratified 80/20 split: predict the test rows, then score them against the truth.
Metrics by task
- Classification: accuracy and ROC-AUC, plus per-class results, against a majority-class baseline.
- Regression: RMSE, NRMSE, R-squared, and interval coverage, against a mean baseline.
- Anomaly detection: precision, recall, and PR-AUC where labels exist; otherwise score separation, clearly marked as unsupervised.
- Imputation: NRMSE on known cells that were hidden on purpose.
An example
Take a churn model scoring 96.1% on 4,480 test rows. Against a 71.0% majority baseline that is a lift of 25.1 points. The weakest slice, customers with under three months of tenure, scores 84%. Each number tells you something the headline hides.
How to read one
- Look at the lift over the baseline, not just the headline.
- Check per-class results and the weakest slice for hidden weak spots.
- Confirm the snapshot and seed, so you know what the number applies to.
- To judge an upgrade, compare two reports and read the delta.
What it does not tell you
A held-out score is an estimate made at creation time, tied to one snapshot. It is not live production accuracy. That only comes from labels that arrive after deployment.