scikit-learn

Overview

scikit-learn is a Python library for classical machine learning. It gives you a consistent set of tools for preprocessing data, fitting models, building pipelines and measuring results. Almost every object follows the same small interface: you call fit to learn, predict or transform to apply, and score to evaluate. Because of this uniformity, you can swap a linear model for a forest, or add a scaling step, by changing one line.

For tabular data, scikit-learn is the baseline toolkit. It covers linear models, trees, ensembles, support vector machines, nearest neighbors, clustering, dimensionality reduction, imputation, feature encoding, model selection and metrics. It does not train deep neural networks or run on GPUs by default, and it is not built for data that does not fit in memory. It is built to be clear, stable and well documented, and that is why it is the first stop for most tabular and classical ML work. It also serves as the common frame for newer tabular foundation models, many of which expose a scikit-learn style interface.

What it is

scikit-learn, often imported as sklearn, is an open source library built on top of NumPy and SciPy. It provides:

• Supervised learning: classification and regression.

• Unsupervised learning: clustering, dimensionality reduction, density estimation, anomaly detection.

• Preprocessing: scaling, encoding, imputation, feature extraction.

• Model selection: cross-validation, hyperparameter search, metrics.

• Pipelines: chains of steps that behave like a single model.

• Utilities: dataset loaders, inspection tools, calibration, and more.

The design in one picture

Imagine a well-run kitchen. Every station has the same kinds of tools and the same way of passing plates. Chopping, boiling and plating are separate stations, but a plate can move through them in any order you choose. scikit-learn is that kitchen. Every station is an object. Every plate is an array. The pass between stations is the fit and transform pattern.

Core object types

Input and output

The standard input is a two-dimensional array of shape (rows, columns), often written X, and a one-dimensional target y. Inputs can be NumPy arrays, pandas data frames, or sparse matrices. Recent versions can also return data frames from transformers, which keeps column names through a pipeline.

What it is not

• It is not a deep learning framework. For neural networks see PyTorch.

• It is not a distributed engine. Most estimators work on one machine in memory.

• It is not an AutoML system, though it has the parts to build one.

• It is not a data frame library. It reads arrays and works well with data frame tools.

Where it came from

The project started in the late 2000s as a Google Summer of Code effort and grew into a community project with contributors from research labs and industry. Its first public releases arrived around 2010. A well-known paper published in 2011 described the library to the research community, and it has been cited very widely since.

Design choices that stuck

Early on, the authors chose a few principles. These are still visible today:

• A consistent API across all estimators.

• Plain data structures: arrays instead of custom containers.

• Sensible defaults, with parameters exposed through constructors.

• Composition through pipelines instead of large monolithic classes.

• Documentation with examples for every major feature.

• A conservative approach to new algorithms: include methods that are well established and widely used.

Growth

Over the years the library added histogram-based gradient boosting, more flexible preprocessing, support for pandas-style column handling, inspection tools such as permutation importance and partial dependence, calibration tools, and improved handling of missing values in several estimators. The project has a public process for deprecations, so code usually keeps working across releases with warnings in advance.

Ecosystem

Many other libraries follow the scikit-learn interface so they can plug into its pipelines and search tools. Gradient boosting libraries such as XGBoost and LightGBM offer wrappers with the same fit and predict methods. Newer tabular foundation models, including TabPFN, also follow the pattern, which makes it easy to compare them with classical models in the same code.

How it works

This section explains the main ideas in the order you meet them in a project: the estimator API, preprocessing, pipelines, evaluation, and search.

Step 1: The estimator API

Every estimator is created with its settings, called hyperparameters. It learns from data with fit. It then produces outputs with predict or transform.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(C=1.0, max_iter=1000)   # settings
model.fit(X_train, y_train)                         # learn
y_pred = model.predict(X_test)                      # apply

Analogy: a model is like a new employee. You set the job description (hyperparameters), train them on past cases (fit), and then give them new cases (predict).

Learned values end with an underscore, such as coef_ or feature_importances_. This is a naming rule that helps you tell settings from learned results.

Step 2: Split data honestly

Before fitting anything, split the data into a training set and a test set. The test set stays untouched until the end. It gives an honest estimate of how the model will behave on new data.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)

stratify=y keeps the class proportions the same in both parts. random_state makes the split repeatable.

Analogy: a teacher keeps a few exam questions secret. If students saw them in class, the exam would not show what they learned.

Step 3: Preprocess by column type

Real tables mix numbers, categories and missing values. Different columns need different treatment. ColumnTransformer lets you apply a different chain of steps to each group of columns.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

num_cols = ["age", "income", "tenure_months"]
cat_cols = ["plan", "region"]

num_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
cat_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])

pre = ColumnTransformer([
    ("num", num_pipe, num_cols),
    ("cat", cat_pipe, cat_cols),
])

Analogy: a mail room. Letters go one way, parcels another, and each gets the right handling before they reach the delivery van.

Step 4: Chain steps in a pipeline

A Pipeline joins preprocessing and the model into a single object. When you call fit, each step is fit in order. When you call predict, the data flows through each step again, using what was learned from training only.

from sklearn.ensemble import RandomForestClassifier

clf = Pipeline([
    ("pre", pre),
    ("model", RandomForestClassifier(n_estimators=300, random_state=0)),
])
clf.fit(X_train, y_train)

The most important benefit is that it prevents data leakage. Scalers, imputers and encoders learn only from training folds, even inside cross-validation. See Random forest for the model used here.

Step 5: Evaluate with cross-validation

One split gives one estimate, which can be noisy. Cross-validation repeats the split several times and averages the scores.

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(clf, X_train, y_train, cv=cv, scoring="roc_auc")
print(scores.mean(), scores.std())

Analogy: instead of one practice exam, you take five different ones and average the marks. Your estimate of skill becomes more reliable.

Step 6: Search for good hyperparameters

GridSearchCV tries every combination in a grid. RandomizedSearchCV samples combinations at random, which is often more efficient. Newer halving searches start many candidates on small data and keep only the best.

from sklearn.model_selection import RandomizedSearchCV

param_dist = {
    "model__n_estimators": [200, 400, 800],
    "model__max_depth": [None, 6, 12, 20],
    "model__min_samples_leaf": [1, 2, 5, 10],
}
search = RandomizedSearchCV(clf, param_dist, n_iter=20, cv=cv,
                            scoring="roc_auc", random_state=0, n_jobs=-1)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)

The double underscore reaches into a pipeline step. model__max_depth means "the max_depth setting of the step named model."

Step 7: Final check on the test set

After choosing the model and settings using only training data, evaluate once on the test set. Report that number. If you keep tuning on the test set, it stops being an honest measure.

Step 8: Save and reuse

A fitted pipeline can be saved and loaded again, with the usual caution that saved model files should only be loaded from trusted sources and with the same library version.

import joblib
joblib.dump(search.best_estimator_, "model.joblib")
loaded = joblib.load("model.joblib")

How common models learn

For related models see SVM, One-class SVM and Isolation forest.

Key concepts and terms

A worked example

The following example builds an end-to-end classifier on a made-up customer table. The data is generated, so the exact numbers will differ on your machine.

Create sample data

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 2000
df = pd.DataFrame({
    "age": rng.integers(18, 80, n),
    "income": rng.normal(50000, 15000, n).round(0),
    "tenure_months": rng.integers(1, 72, n),
    "plan": rng.choice(["basic", "plus", "pro"], n, p=[0.5, 0.3, 0.2]),
    "region": rng.choice(["north", "south", "east", "west"], n),
})
# introduce some missing values
df.loc[rng.choice(n, 120, replace=False), "income"] = np.nan
df.loc[rng.choice(n, 80, replace=False), "plan"] = None

# a synthetic target that depends on a few columns
logit = -2 + 0.00003 * df["income"].fillna(50000) - 0.02 * df["tenure_months"] \
        + (df["plan"] == "basic") * 0.8
p = 1 / (1 + np.exp(-logit))
df["churned"] = (rng.random(n) < p).astype(int)

X = df.drop(columns="churned")
y = df["churned"]

Build and compare models

from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0)

pre_linear = ColumnTransformer([
    ("num", num_pipe, num_cols),
    ("cat", cat_pipe, cat_cols),
])
# tree models do not need scaling
pre_tree = ColumnTransformer([
    ("num", SimpleImputer(strategy="median"), num_cols),
    ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                      ("oh", OneHotEncoder(handle_unknown="ignore"))]), cat_cols),
])

candidates = {
    "logreg": Pipeline([("pre", pre_linear), ("m", LogisticRegression(max_iter=1000))]),
    "forest": Pipeline([("pre", pre_tree), ("m", RandomForestClassifier(300, random_state=0))]),
    "hgb":    Pipeline([("pre", pre_tree), ("m", HistGradientBoostingClassifier(random_state=0))]),
}

cv = StratifiedKFold(5, shuffle=True, random_state=0)
for name, pipe in candidates.items():
    s = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="roc_auc")
    print(f"{name:8s} AUC {s.mean():.3f} +/- {s.std():.3f}")

Read the output

You will see one line per model, with a mean and a spread. Compare the means, but also the spread. If two models differ by less than the spread, you cannot say one is better. Pick the simpler model in that case.

Final evaluation and inspection

from sklearn.metrics import roc_auc_score, classification_report
from sklearn.inspection import permutation_importance

best = candidates["hgb"].fit(X_train, y_train)
proba = best.predict_proba(X_test)[:, 1]
print("Test AUC:", roc_auc_score(y_test, proba))
print(classification_report(y_test, best.predict(X_test)))

imp = permutation_importance(best, X_test, y_test, n_repeats=10,
                             random_state=0, scoring="roc_auc")
for i in imp.importances_mean.argsort()[::-1]:
    print(X.columns[i], round(imp.importances_mean[i], 4))

Permutation importance shuffles one column at a time and measures the drop in score. A big drop means the model relies on that column.

A tiny numeric walk-through of scaling

Suppose income has a mean of 50,000 and a standard deviation of 15,000. A value of 65,000 becomes (65,000 minus 50,000) divided by 15,000, which is 1.0. A value of 35,000 becomes minus 1.0. After scaling, all numeric columns have about zero mean and unit spread, so no column dominates a distance-based or regularized model only because of its units.

A tiny numeric walk-through of one-hot encoding

The column plan has three values: basic, plus, pro. One-hot encoding turns one column into three. A row with plus becomes (0, 1, 0). A row with a missing plan that was imputed to the most frequent value, basic, becomes (1, 0, 0). Models that need numbers can now use the category without assuming that pro is "three times" basic.

Strengths and limitations

Strengths

• A uniform API that makes models interchangeable.

• Pipelines and column transformers that prevent leakage and keep code tidy.

• Broad coverage of classical algorithms with good defaults.

• Strong documentation, user guide and examples.

• Stable releases with a clear deprecation policy.

• Good tools for evaluation, inspection and calibration.

• Works well with NumPy, pandas, SciPy and plotting libraries.

• Easy to learn, so it is a common teaching tool.

Limitations

• Mostly CPU and in-memory. Very large data needs other tools or sampling.

• No built-in deep learning. Use PyTorch or similar for that.

• Limited support for streaming and out-of-core learning, available only in some estimators.

• Not every estimator handles missing values or categorical features natively. Check the documentation for each one.

• Pipelines return arrays by default in some steps, which can lose column names unless configured.

• Hyperparameter search can be slow without careful budget control.

• Not designed for model serving. You need extra code or tools to deploy.

• Feature engineering is manual. The library gives the blocks, not the ideas.

How it compares with alternatives

A common workflow: start with scikit-learn for a fast baseline and clean pipeline, then try a boosting library or a foundation model with the same split and metrics, and keep the best. The scikit-learn pipeline stays as the frame for data preparation and evaluation.

Role in tabular foundation models

Tabular foundation models are large models pre-trained on many tables so they can predict on a new table with little or no training. scikit-learn matters here in three ways: as the baseline, as the interface, and as the toolbox around the model.

Row and column structure

scikit-learn expects a matrix with rows as samples and columns as features. This is the same shape that tabular foundation models read. A foundation model treats each row as an example and each column as a feature, and usually does not assume that columns have a fixed order. A scikit-learn style wrapper hides this and gives you fit(X, y) and predict(X). In many foundation models, fit does not train weights. It stores the training rows as context, and predict runs one forward pass that reads the context and the query rows together. See TabPFN and /model/architecture.

Mixed types and encoding

Foundation models need numeric input of some form. Preprocessing with ColumnTransformer handles dates, categories and numbers in a clear and tested way. Some foundation models have their own internal encoders and need little preprocessing. Others work better when high-cardinality categories are encoded first. A scikit-learn pipeline lets you try both without changing the rest of your code.

Missing values

scikit-learn offers SimpleImputer, KNNImputer and IterativeImputer, plus indicators that flag missingness. These are the standard baselines against which learned imputation is judged. Foundation models trained on tables with missing cells can often handle gaps directly, and a model-based approach can fill cells by predicting each column from the others. See Imputation networks. When you benchmark, compare against the simple imputers inside a pipeline so the results are fair.

No column names

Some tabular foundation models are designed to work without column names, so that the same weights apply to any table. In scikit-learn, arrays have no names by default, which fits this design. When you do have names, you can keep them for reporting while passing only values to the model.

Pre-training, synthetic and real data

Foundation models are pre-trained on large numbers of synthetic or real tables. scikit-learn is often used in the data generation step, for example to sample random regression and classification problems, to build random feature interactions, or to produce noisy variants of tables. Its dataset generators and preprocessing tools are convenient building blocks. See Synthetic data generation.

In-context learning

In-context learning means the model uses labeled rows given at inference time as its training signal, with no weight updates. The scikit-learn API fits this neatly. Calling fit stores the rows. Calling predict triggers inference. Cross-validation, grid search and calibration wrappers all work on top of this pattern, so you can evaluate a foundation model with exactly the same code as a random forest.

Continual learning and updating

Classical estimators in scikit-learn that support partial_fit can learn from batches of data over time. Foundation models handle new data differently: you can simply give them a new context. When you need to adapt the weights, the model is fine-tuned with a deep learning framework instead. Pipelines remain useful in both cases because they keep preprocessing consistent between old and new data. See Continued pre-training.

Anomaly detection

scikit-learn includes several anomaly detectors, including Isolation forest, One-class SVM, local outlier factor and elliptic envelope. They are the standard comparison points for any new method. A foundation model can be used for anomaly detection by scoring how unlikely a row is given the rest of the table. The classical detectors provide the baseline numbers. See Anomaly detection.

Semantic and sector classification

When the task is to classify rows into categories based on text or code fields, scikit-learn's text tools, such as TF-IDF vectorizers, can build a strong baseline that is easy to explain. A foundation model that reads both text and numbers may do better, and the baseline shows how much better. For evidence and comparisons see /model/benchmarks and /model/research.

Practical guidance and best practices

Always use pipelines

Put every preprocessing step inside a pipeline. Do not scale or impute the whole data frame before splitting. That leaks information and inflates scores.

Match preprocessing to the model

Pick metrics that match the goal

• Use ROC AUC or average precision for ranking.

• Use F1 or recall at a chosen precision when classes are imbalanced.

• Use log loss or Brier score when probabilities matter.

• Use mean absolute error or R squared for regression, and look at residual plots.

• Use a time-aware split when data has a time order.

Handle imbalance

Start with class_weight="balanced" in models that support it. Use stratified splits. Choose a threshold from the precision and recall curve instead of using 0.5. Be careful with oversampling inside cross-validation: it must happen inside the training fold only.

Control randomness

Set random_state for splits and for models. Run several seeds for important comparisons and report the mean and spread.

Use sensible search budgets

Start with a coarse random search, look at the results, and refine the ranges. Use fewer folds on large data. Use halving searches to save time.

Inspect, do not just score

• Check feature importance with permutation importance on held-out data.

• Plot partial dependence for a few key features.

• Look at errors by group, such as region or plan.

• Check calibration with a reliability curve.

Keep versions consistent

Pin the library version in your project. Save the version with every saved model. Test loading in a clean environment before relying on it.

A baseline-first routine

1. Write a pipeline with a dummy model that predicts the majority class or the mean.

2. Add a linear model.

3. Add a tree ensemble or histogram gradient boosting.

4. Try a boosting library or a foundation model in the same pipeline.

5. Compare under the same folds and metrics.

6. Keep the simplest model whose score is within noise of the best.

Common mistakes

1. Fitting a scaler or imputer on the full data before splitting.

2. Tuning on the test set until the score looks good.

3. Using accuracy on a heavily imbalanced problem.

4. Ignoring the spread of cross-validation scores.

5. Random splits on time-ordered data, which leak the future.

6. Splitting rows from the same customer or device into both train and test sets.

7. One-hot encoding a column with thousands of values without thinking about memory.

8. Forgetting handle_unknown="ignore" so that new categories crash prediction.

9. Treating ordinal-encoded categories as numbers with meaningful distances.

10. Comparing models with different preprocessing or different folds.

11. Loading saved models from untrusted sources or across incompatible versions.

12. Using default settings for a distance-based model without scaling.

13. Reading feature importance from a model that was overfit.

14. Over-trusting impurity-based importance, which can favor high-cardinality features.

Frequently asked questions

What is scikit-learn used for?

scikit-learn is used to build classical machine learning models on tabular data. You use it to clean and encode features, train classifiers and regressors, group rows with clustering, reduce dimensions, detect anomalies and measure results with cross-validation. It is the standard baseline toolkit in Python for data that fits in memory.

Is scikit-learn good for deep learning?

No, it is not meant for that. It includes a basic multi-layer perceptron, but for modern neural networks, GPUs and custom architectures you should use a deep learning framework such as PyTorch. Many projects use scikit-learn for preprocessing and evaluation and a deep learning framework for the model itself.

What is the difference between fit, transform and predict?

fit learns something from data, such as the mean of a column or the weights of a model. transform applies a learned change to data, such as scaling. predict produces model outputs for new rows. Transformers have fit and transform, and predictors have fit and predict. A pipeline calls the right method at each step.

Why should I use a pipeline?

A pipeline keeps all steps together, ensures the same preparation is applied at training and prediction time, and prevents data leakage during cross-validation. It also makes the whole workflow a single object that you can tune, save and reuse. Without it, it is easy to scale or impute using information from the test set by accident.

How does scikit-learn handle missing values?

It provides imputers that fill gaps with a statistic such as the median or the most frequent value, a nearest-neighbor imputer, and an iterative imputer that models each column from the others. Some estimators, such as histogram gradient boosting, accept missing values directly. Adding a missing indicator can also help when missingness itself carries a signal.

How do I choose between scikit-learn models and gradient boosting libraries?

Start with scikit-learn for a fast baseline and clean evaluation. Histogram gradient boosting inside scikit-learn is already strong on many tables. If you need more speed on very large data, more tuning options or GPU support, try a dedicated boosting library through its scikit-learn style wrapper, and compare using the same folds.

Does scikit-learn work with tabular foundation models?

Yes, in practice. Many tabular foundation models offer a scikit-learn style interface with fit and predict, so they can sit inside pipelines, cross-validation and search tools. In those models, fit often stores the training rows as context rather than updating weights. scikit-learn then provides the preprocessing, splitting and metrics around the model.

How many cross-validation folds should I use?

Five or ten folds are common. Use more folds for small data to get a lower-bias estimate, and fewer for large data to save time. Always use stratified folds for classification and a time-aware splitter for ordered data. Repeat with different seeds if the decision is important.

Can scikit-learn handle large datasets?

It can handle moderately large tables on a single machine, especially with sparse matrices and histogram-based models. For data that does not fit in memory, you can sample, use estimators with partial_fit, or move to a distributed or out-of-core tool. Choosing the right tool depends on the size of the table and the model.

How do I know if a model is overfitting?

Compare training and validation scores. If the training score is much higher than the validation score, the model is memorizing noise. Learning curves show how scores change with the amount of data. Regularization, simpler models, more data, early stopping and cross-validated tuning all help reduce overfitting.

Glossary

• Cross-validation: repeated splitting of data to estimate performance.

• Data leakage: use of information at training time that would not be available at prediction time.

• Estimator: an object that learns from data.

• Feature: an input column.

• Hyperparameter: a setting chosen before training.

• Imputation: filling in missing values.

• One-hot encoding: turning a category into several yes or no columns.

• Pipeline: a chain of preprocessing steps and a model.

• Regularization: a penalty that discourages complex models.

• Stratified split: a split that keeps class proportions.

• Target: the value to predict.

• Transformer: an object that changes data.

Summary

scikit-learn is the standard Python toolkit for classical machine learning. Its strength is a consistent estimator API, pipelines that prevent leakage, solid evaluation tools and broad coverage of tabular algorithms. It is limited to mostly in-memory CPU work, and it is not a deep learning framework, but it remains the default starting point and the common frame for comparing models.

For tabular foundation models, scikit-learn supplies the baselines, the preprocessing, the evaluation and the interface. A model that stores context rows in fit and predicts in one pass fits naturally into its pipelines, cross-validation and search tools. Use it to build leak-free baselines, test imputation and anomaly detection methods against standard choices, and keep comparisons fair. For related reading see XGBoost, LightGBM, Random forest, SVM, Isolation forest, TabPFN, /model/architecture, /model/research and /model/benchmarks.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: