Classical machine learning models: regression, trees, boosting, clustering and PCA
Before deep learning, a small set of algorithms solved most prediction problems, and on tables many still do. This page compares them and says when each fits.
Overview
Classical models are fast, need little data and are easy to explain. For tabular business data, gradient-boosted trees remain a strong default, and simple models are the baseline every new method must beat.
Supervised models
| Model | Idea | Strengths | Weaknesses | Read more |
|---|---|---|---|---|
| Linear regression | Weighted sum of features | Simple, interpretable | Straight-line relations only | Page, simulation |
| Logistic regression | Linear model for probabilities | Calibrated baseline | Needs feature engineering | Page |
| Decision tree | Series of if-then splits | Readable, handles mixed data | Overfits alone | Page, simulation |
| Random forest | Many trees on random subsets | Robust, little tuning | Larger, slower | Page |
| Gradient boosting | Trees added to fix errors | Top accuracy on tables | Needs tuning | XGBoost, LightGBM, CatBoost |
| k-nearest neighbours | Copy the closest examples | No training | Slow, sensitive to scale | Page |
| Support vector machine | Widest margin between classes | Strong on small data | Scales poorly | Page |
| Naive Bayes | Probabilities assuming independence | Very fast, good for text | Strong assumption | Page |
Unsupervised models
| Model | Purpose | Read more |
|---|---|---|
| k-means | Group similar rows | Clustering, simulation |
| PCA | Compress features into a few directions | PCA, simulation |
| t-SNE, UMAP | Visualise high-dimensional data | t-SNE, UMAP |
| Isolation forest | Find anomalies | Isolation forest |
Choosing
Start with a linear model or a small tree as a baseline. For accuracy on tables, tune gradient boosting with Optuna or try a tabular foundation model, which needs no tuning. Use SHAP to explain any of them. scikit-learn implements almost all classical models with one interface.
Frequently asked questions
Why do trees beat neural networks on tables?
Tables mix scales and types, have irregular feature effects and are often small; trees handle all three without preprocessing. Tabular foundation models are the first neural approach to beat them consistently on small and medium tables.
Is random forest or gradient boosting better?
Gradient boosting is usually more accurate after tuning; random forest is more forgiving with defaults.
Summary
Linear models for baselines and explanation, trees and boosting for accuracy on tables, clustering and PCA for structure. Know these well before reaching for deep learning.
Back to the topic
This page is part of Machine learning, which maps every concept, tool, simulation and article on the subject.