Data preprocessing

Overview

Data preprocessing is everything you do to raw data before a model can learn from it. Real datasets have empty cells, numbers on wildly different scales, text categories such as "red" and "blue", extreme values from sensor glitches, and classes where one outcome is a hundred times rarer than the other. Most models cannot accept missing values or strings at all, and many behave badly when features are on different scales. Preprocessing fixes these problems in a consistent, repeatable way.

It is also where many evaluation mistakes happen. If you compute a mean for imputation, a scaling factor or a category list on the whole dataset before splitting it, information from the test data leaks into training, and your scores become optimistic. The professional habit is to treat every preprocessing step as part of the model: it is fitted on training data only, then applied unchanged to validation, test and production data. In scikit-learn, Pipeline and ColumnTransformer make this habit easy.

An everyday analogy is preparing ingredients before cooking. You wash vegetables, cut them to similar sizes so they cook evenly, replace a missing ingredient with a sensible substitute and set aside anything spoiled. A good cook does this the same way every time, following the recipe card, not by tasting the guests' plates first. Preprocessing with pipelines is that recipe card for data.

What it is

Preprocessing sits just before feature engineering, and the two overlap. A useful rule of thumb: preprocessing makes the existing data usable and consistent; feature engineering creates new information. Both should run inside the same pipeline so that training and production stay identical.

How much preprocessing you need depends on the model. Linear models, support vector machines, k-nearest neighbours and neural networks are sensitive to scale and need numeric inputs. Tree models such as random forests and gradient boosting ignore scale, and some, such as LightGBM, CatBoost and scikit-learn's histogram gradient boosting, handle missing values and categories natively.

Where it came from

The statistical study of missing data was put on a formal footing by Donald Rubin's 1976 paper "Inference and missing data" in Biometrika, which defined the conditions now called missing completely at random, missing at random and missing not at random. Multiple imputation followed in Rubin's later work, and the chained-equations approach was popularised by the R package mice, described by Stef van Buuren and Karin Groothuis-Oudshoorn in the Journal of Statistical Software in 2011. scikit-learn's IterativeImputer cites that work.

Box-plot fences for outliers come from John Tukey's exploratory data analysis. For class imbalance, Nitesh Chawla and colleagues introduced SMOTE, the synthetic minority over-sampling technique, in the Journal of Artificial Intelligence Research in 2002, and the imbalanced-learn library, described by Guillaume Lemaître and colleagues in the Journal of Machine Learning Research in 2017, brought resampling methods into scikit-learn-style pipelines. Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou introduced isolation forests for outlier detection in 2008.

On the software side, scikit-learn's Pipeline has long been the standard way to chain preprocessing and models. ColumnTransformer, which applies different steps to different columns, made mixed-type tables straightforward. Later releases added conveniences such as infrequent-category handling in OneHotEncoder (version 1.1), the sparse_output name (version 1.2), TargetEncoder (version 1.3) and native categorical support by default in histogram gradient boosting (version 1.6).

How it works

Missing values and imputation

First, understand why data is missing. Under missing completely at random (MCAR), missingness is unrelated to anything; under missing at random (MAR), it depends on other observed columns; under missing not at random (MNAR), it depends on the missing value itself, such as high earners leaving income blank. Dropping rows is only safe under MCAR and when few rows are affected.

• Simple imputation fills with the mean, median, most frequent value or a constant. The median is robust to skew. It is fast and works well with tree models.
• Missing indicators add a 0/1 column saying "this value was missing". Under MNAR, the fact of missingness can be predictive. SimpleImputer(add_indicator=True) adds them automatically.
• KNN imputation (KNNImputer) fills a value from the most similar rows, which preserves local structure but is slower and sensitive to scale.
• Iterative imputation (IterativeImputer) models each feature with missing values from the others in a round-robin, following the chained-equations idea. In current stable scikit-learn releases it is still experimental and must be enabled with from sklearn.experimental import enable_iterative_imputer.
• Model-based and learnt imputation, such as imputation networks and masked cell modelling, learn joint patterns across columns and can work well on large tables.

Whatever the method, fit the imputer on training data only.

Scaling

Standard:  z = (x − μ) / σ
Min-max:   x′ = (x − min) / (max − min)
Robust:    x′ = (x − median) / IQR,   IQR = Q3 − Q1
Balanced class weight:  w_c = N / (K · N_c)

Here x is a raw value; μ and σ are the training mean and standard deviation of that feature; min and max are its training minimum and maximum; the median, Q1 and Q3 are its training median and 25th and 75th percentiles, and IQR is their spread; N is the number of training rows, K the number of classes and N_c the number of rows in class c, so w_c is the weight given to each example of class c.

Standard scaling centres a feature on zero with unit variance; it suits linear models, SVMs, PCA and neural networks. Min-max scaling maps to the range 0 to 1, which suits bounded inputs such as pixel intensities, but one extreme value squashes everything else. Robust scaling uses the median and interquartile range, so outliers barely affect it; scikit-learn's RobustScaler uses the 25th and 75th percentiles by default. For heavily skewed data, a power or quantile transform before scaling often helps. Tree models do not need scaling at all.

Encoding categoricals

• One-hot encoding creates a 0/1 column per category. It makes no assumption about order and suits linear models and neural networks. In scikit-learn, OneHotEncoder raises an error on unseen categories by default; handle_unknown="ignore" encodes them as all zeros, and min_frequency or max_categories group rare values into an "infrequent" column.
• Ordinal encoding maps categories to integers. It is right for genuinely ordered categories such as "small, medium, large", and works well with tree models even for unordered ones, but misleads linear models.
• Target encoding replaces each category with a smoothed average of the target. It is compact for high-cardinality columns, but must be cross-fitted to avoid leakage; scikit-learn's TargetEncoder does this in fit_transform.
• Native handling: CatBoost, LightGBM and histogram gradient boosting can take categorical columns directly.

Outliers

An outlier may be an error (a negative age, a temperature of 999 from a sensor fault) or a real but rare case (a very large order). Treat them differently. Fix or remove clear errors after checking the source. For real extremes, prefer methods that reduce their influence without deleting them: clipping (winsorising) at chosen percentiles, log or power transforms, robust scaling, or robust losses. A common screening rule flags values more than 1.5 interquartile ranges beyond the quartiles, following Tukey's box plots. For multivariate outliers, isolation forests and other anomaly detection methods help. Decide outlier rules on training data and apply the same rules later.

Class imbalance

When one class is rare, a model can score well by mostly ignoring it. Options, in rough order of preference:

1. Use the right metric and threshold. Judge models with precision-recall measures and choose the decision threshold from costs; see evaluation metrics. Often this alone is enough.
2. Class weights. Make each minority example count more in the loss. class_weight="balanced" in scikit-learn uses the formula above. It changes no data and works with most models.
3. Resampling. Random under-sampling drops majority rows; random over-sampling repeats minority rows; SMOTE creates synthetic minority examples by interpolating between neighbours. Resample only the training folds, never validation or test data; imbalanced-learn's own Pipeline applies samplers during fitting only.

Weights and resampling distort predicted probabilities, so recalibrate if you need them as probabilities.

Train, validation and test splits

Split before you look closely at the data. A common pattern is training data for fitting, a validation set or cross-validation for model selection and tuning, and a test set touched once at the end. Choose the split to match how the model will be used:

• Stratified splits keep class proportions equal in each part, important for imbalanced data.
• Group splits keep all rows of one customer, patient or machine in the same part, so the model is tested on new entities (GroupKFold).
• Time-based splits train on the past and test on the future (TimeSeriesSplit), essential for forecasting and any data that drifts.

Leakage happens when information unavailable at prediction time reaches training: preprocessing fitted on all data, duplicates across splits, features computed from the future, or the same entity on both sides. The cure is to split first, fit every step on training data only, and wrap everything in a pipeline.

Pipelines in scikit-learn

A Pipeline chains steps so that fit fits each transformer on the training data and passes the result on, and predict applies the fitted transformers and the model in order. A ColumnTransformer routes different columns to different sub-pipelines. Because cross-validation and hyperparameter search refit the whole pipeline on each training fold, leakage from preprocessing disappears automatically.

from sklearn.compose import ColumnTransformer, make_column_selector as cols
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                    ("scale", StandardScaler())])
categorical = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                        ("onehot", OneHotEncoder(handle_unknown="ignore", min_frequency=20))])

prep = ColumnTransformer([("num", numeric, cols(dtype_include="number")),
                          ("cat", categorical, cols(dtype_include=["object", "category"]))])
model = Pipeline([("prep", prep),
                  ("clf", LogisticRegression(class_weight="balanced", max_iter=2000))])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
print(cross_val_score(model, X_train, y_train, cv=cv, scoring="average_precision").mean())

The fitted model can be saved and used in production as one object, so the same medians, scales and category lists are applied to new data.

Key concepts and terms

A worked example

This is an illustrative scenario. A bank predicts loan default from 40,000 past applications with 25 columns: income, age, loan amount, employment type, region, number of previous loans and more. Defaults are 4 per cent of rows.

Inspection. Income is missing in 9 per cent of rows, more often for self-employed applicants, so it is probably not missing completely at random. Income is heavily right-skewed with a few values above ten million, two of which turn out to be data-entry errors. Region has 300 values, 220 of them with fewer than 20 applications.

Splitting. The team holds out the most recent six months as a test set, because the model will score future applications, and uses stratified five-fold cross-validation on the rest.

A leaky first attempt. A notebook imputes income with the overall median and standardises every column on the full dataset, then splits. Cross-validated average precision is 0.31, and the test score is 0.27.

The pipeline version. Median imputation with a missing indicator, a log transform then standard scaling for income, one-hot encoding with min_frequency=20 for region, and class_weight="balanced" in logistic regression, all inside one pipeline. Cross-validated average precision is 0.29 and the test score 0.28: lower in cross-validation than the leaky version, but honest, and now matching the test set. The missing-income indicator turns out to be one of the stronger predictors.

A comparison. A histogram gradient-boosting model on the same split, with native missing-value and categorical handling and no scaling, reaches a test average precision of 0.34 with far less preprocessing code.

Strengths and limitations

Strengths:

• Makes data usable by models that need complete, numeric, well-scaled input.
• Pipelines prevent leakage and keep training and production consistent.
• Good imputation and missing indicators can recover real signal from gaps.
• Handling imbalance and outliers carefully improves the cases that matter most.
• Mostly cheap to compute and easy to automate.

Limitations:

• Choices such as imputation method or outlier rules can bias results if made without understanding the data.
• Simple imputation shrinks variance and can weaken relationships between features.
• Resampling and class weights distort probabilities.
• One-hot encoding of very high-cardinality columns creates huge sparse matrices.
• No preprocessing can fix data that is unrepresentative, mislabelled or missing the key information.

How it compares with alternatives

Practical guidance and best practices

1. Split the data first, by time or group if that matches deployment, and keep a test set untouched.
2. Put every fitted step, including imputation, scaling and encoding, inside a pipeline.
3. Investigate why values are missing before choosing an imputation method, and add missing indicators when missingness may carry signal.
4. Prefer the median and robust scaling for skewed or outlier-prone features.
5. Set handle_unknown in encoders so that new categories in production do not crash the model.
6. Handle imbalance first with metrics and thresholds, then class weights, and only then resampling inside the training folds.
7. Check for duplicate rows and repeated entities across splits.
8. Save the fitted pipeline as a single artefact and version it with the model.
9. Monitor input distributions and missing rates in production, since changes there often explain sudden drops in quality.

Common mistakes

• Scaling, imputing or encoding on the full dataset before splitting.
• Using random splits for time-dependent data, so the model is tested on the past it already saw.
• Oversampling before cross-validation, so copies of the same minority row appear in both training and validation folds.
• Deleting all rows with any missing value and silently changing the population.
• Applying min-max scaling to data with extreme outliers.
• Ordinal-encoding unordered categories for a linear model.
• Fitting the preprocessing again on production data instead of reusing the training pipeline.

Frequently asked questions

Do I need to scale features for tree-based models?

No. Decision trees, random forests and gradient boosting split on thresholds, so any monotonic rescaling leaves them unchanged. Scaling does matter for distance-based models, linear models with penalties, SVMs, PCA and neural networks. If a pipeline mixes models, scaling does no harm to the trees.

Which imputation method should I use?

Start with the median for numbers and the most frequent value for categories, plus missing indicators, and compare against a model with native missing-value support. Move to KNN or iterative imputation when features are strongly related and the missing share is substantial. Always compare methods by validation performance, not by how plausible the filled values look.

Should I remove outliers?

Remove or correct values that are clearly errors, after checking the source. Keep genuine extremes, but limit their influence with clipping, transforms, robust scaling or robust losses. Removing real rare cases can make a model fail exactly where it is needed, such as large fraudulent transactions.

Is SMOTE better than class weights?

Not reliably. Class weights are simpler, change no data and often perform as well. SMOTE can help some models on some datasets, but it can also create unrealistic examples, especially with categorical or high-dimensional data. Try weights and threshold tuning first, and evaluate any resampling inside cross-validation.

What is the difference between a validation set and a test set?

The validation set, or cross-validation, is used to make decisions: choosing models, features and hyperparameters. The test set is used once, at the end, to estimate performance on new data. If you use the test set to make decisions, it becomes a second validation set and no longer gives an unbiased estimate.

Why use ColumnTransformer instead of transforming columns by hand?

It keeps the column routing inside the model object, so it is fitted on training folds only, applied consistently to new data and saved with the model. Hand-written transformations in a notebook are easy to apply differently in production and easy to fit on the full dataset by mistake.

Glossary

• Imputer: a component that fills missing values.
• KNN imputation: filling a value from the most similar complete rows.
• Iterative imputation: modelling each incomplete feature from the others in turns.
• Min-max scaling: mapping values to a fixed range, usually 0 to 1.
• Winsorising: capping extreme values at chosen percentiles.
• Under-sampling: removing majority-class rows to balance classes.
• Over-sampling: repeating or synthesising minority-class rows.
• Group split: a split that keeps all rows of an entity together.
• Time-based split: training on earlier data and testing on later data.
• ColumnTransformer: a scikit-learn tool that applies different transformations to different columns.

Summary

Data preprocessing turns raw, messy data into clean, consistent input for a model. It covers missing values (understanding why they are missing, imputing with simple, KNN or iterative methods and adding missing indicators), scaling (standard, min-max and robust), encoding categories (one-hot, ordinal and cross-fitted target encoding), handling outliers and class imbalance (metrics and thresholds first, then class weights, then resampling inside training folds), and splitting data in a way that matches deployment. The single most important habit is to fit every step on training data only, inside a scikit-learn Pipeline with a ColumnTransformer, so that cross-validation stays honest and production applies exactly the same transformations as training.

Related

It belongs to Training fundamentals in my stack, under Machine learning. Others in the same group: