Özer Cem KELAHMETKELAHMET
Topics · Machine learning

Classical machine learning models: regression, trees, boosting, clustering and PCA

Before deep learning, a small set of algorithms solved most prediction problems, and on tables many still do. This page compares them and says when each fits.

Overview

Classical models are fast, need little data and are easy to explain. For tabular business data, gradient-boosted trees remain a strong default, and simple models are the baseline every new method must beat.

Supervised models

Model Idea Strengths Weaknesses Read more
Linear regression Weighted sum of features Simple, interpretable Straight-line relations only Page, simulation
Logistic regression Linear model for probabilities Calibrated baseline Needs feature engineering Page
Decision tree Series of if-then splits Readable, handles mixed data Overfits alone Page, simulation
Random forest Many trees on random subsets Robust, little tuning Larger, slower Page
Gradient boosting Trees added to fix errors Top accuracy on tables Needs tuning XGBoost, LightGBM, CatBoost
k-nearest neighbours Copy the closest examples No training Slow, sensitive to scale Page
Support vector machine Widest margin between classes Strong on small data Scales poorly Page
Naive Bayes Probabilities assuming independence Very fast, good for text Strong assumption Page

Unsupervised models

Model Purpose Read more
k-means Group similar rows Clustering, simulation
PCA Compress features into a few directions PCA, simulation
t-SNE, UMAP Visualise high-dimensional data t-SNE, UMAP
Isolation forest Find anomalies Isolation forest

Choosing

Start with a linear model or a small tree as a baseline. For accuracy on tables, tune gradient boosting with Optuna or try a tabular foundation model, which needs no tuning. Use SHAP to explain any of them. scikit-learn implements almost all classical models with one interface.

Frequently asked questions

Why do trees beat neural networks on tables?

Tables mix scales and types, have irregular feature effects and are often small; trees handle all three without preprocessing. Tabular foundation models are the first neural approach to beat them consistently on small and medium tables.

Is random forest or gradient boosting better?

Gradient boosting is usually more accurate after tuning; random forest is more forgiving with defaults.

Summary

Linear models for baselines and explanation, trees and boosting for accuracy on tables, clustering and PCA for structure. Know these well before reaching for deep learning.

Back to the topic

This page is part of Machine learning, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open