# Classical machine learning models: regression, trees, boosting, clustering and PCA

Before deep learning, a small set of algorithms solved most prediction problems, and on tables many still do. This page compares them and says when each fits.

## Overview

Classical models are fast, need little data and are easy to explain. For tabular business data, gradient-boosted trees remain a strong default, and simple models are the baseline every new method must beat.

## Supervised models

| Model | Idea | Strengths | Weaknesses | Read more |
|---|---|---|---|---|
| Linear regression | Weighted sum of features | Simple, interpretable | Straight-line relations only | [Page](/tools/linear-regression), [simulation](/ai-simulations/linear-regression) |
| Logistic regression | Linear model for probabilities | Calibrated baseline | Needs feature engineering | [Page](/tools/logistic-regression) |
| Decision tree | Series of if-then splits | Readable, handles mixed data | Overfits alone | [Page](/tools/decision-trees), [simulation](/ai-simulations/decision-tree) |
| Random forest | Many trees on random subsets | Robust, little tuning | Larger, slower | [Page](/tools/random-forest) |
| Gradient boosting | Trees added to fix errors | Top accuracy on tables | Needs tuning | [XGBoost](/tools/xgboost), [LightGBM](/tools/lightgbm), [CatBoost](/tools/catboost) |
| k-nearest neighbours | Copy the closest examples | No training | Slow, sensitive to scale | [Page](/tools/k-nearest-neighbors) |
| Support vector machine | Widest margin between classes | Strong on small data | Scales poorly | [Page](/tools/svm) |
| Naive Bayes | Probabilities assuming independence | Very fast, good for text | Strong assumption | [Page](/tools/naive-bayes) |

## Unsupervised models

| Model | Purpose | Read more |
|---|---|---|
| k-means | Group similar rows | [Clustering](/tools/clustering), [simulation](/ai-simulations/k-means-clustering) |
| PCA | Compress features into a few directions | [PCA](/tools/principal-component-analysis), [simulation](/ai-simulations/pca) |
| t-SNE, UMAP | Visualise high-dimensional data | [t-SNE](/tools/t-sne), [UMAP](/tools/umap) |
| Isolation forest | Find anomalies | [Isolation forest](/tools/isolation-forest) |

## Choosing

Start with a linear model or a small tree as a baseline. For accuracy on tables, tune gradient boosting with [Optuna](/tools/optuna) or try a [tabular foundation model](/topics/machine-learning/tabular-foundation-models), which needs no tuning. Use [SHAP](/tools/shap) to explain any of them. [scikit-learn](/tools/scikit-learn) implements almost all classical models with one interface.

## Frequently asked questions

### Why do trees beat neural networks on tables?

Tables mix scales and types, have irregular feature effects and are often small; trees handle all three without preprocessing. Tabular foundation models are the first neural approach to beat them consistently on small and medium tables.

### Is random forest or gradient boosting better?

Gradient boosting is usually more accurate after tuning; random forest is more forgiving with defaults.

## Summary

Linear models for baselines and explanation, trees and boosting for accuracy on tables, clustering and PCA for structure. Know these well before reaching for deep learning.

## Back to the topic

This page is part of **[Machine learning](/topics/machine-learning)**, which maps every concept, tool, simulation and article on the subject.
