# Feature engineering Feature engineering turns raw data into inputs a model can learn from. Learn transforms, date and text features, target encoding, aggregations and selection. ## Overview Feature engineering is the work of turning raw data into the input columns, called features, that a model learns from. A raw table might hold a timestamp, a customer ID, a free-text comment and a list of past orders. A model cannot do much with those directly. Feature engineering turns them into "hour of day", "is a weekend", "number of orders in the last 30 days", "average basket size", "share of orders returned" and "comment mentions delivery". The model then sees the pattern instead of having to discover it from scratch. For tabular data, good features are often worth more than a better algorithm. A plain [logistic regression](/tools/logistic-regression) with well-chosen features can beat a complex model fed raw columns. Feature engineering is also where domain knowledge enters the model: an analyst who knows that payday matters for spending, or that machines wear faster after a heatwave, can encode that knowledge as a feature. The flip side is risk. Badly built features, especially those that peek at the target or the future, produce models that look brilliant in testing and fail in production. An everyday analogy: a doctor does not diagnose from a patient's raw height and weight. They compute body-mass index, look at the change in weight since the last visit and compare blood pressure with the normal range for the patient's age. Each of these is an engineered feature: a combination of raw facts that makes the important signal easy to see. ## What it is | Family | Examples | What it helps with | |---|---|---| | Numeric transforms | Log, square root, power transforms, ratios, differences | Skewed values, multiplicative effects | | Binning | Equal-width, quantile, domain thresholds | Non-linear effects in linear models, robustness to outliers | | Interactions | Products, ratios, polynomial terms, crosses of categories | Effects that depend on combinations of features | | Date and time | Hour, weekday, month, holiday flags, time since event, cyclic encodings | Seasonality and recency | | Text | Counts, TF-IDF, keyword flags, embeddings | Turning free text into numbers | | Categorical encodings | One-hot, ordinal, frequency, target encoding, hashing | Making categories usable | | Aggregations | Counts, sums, means, trends over related rows and time windows | Relational and event data | | Feature selection | Filters, wrappers, embedded methods | Removing noise and redundancy | Feature engineering sits between [data preprocessing](/tools/data-preprocessing), which cleans and standardises the data, and modelling. The line is blurry: scaling and one-hot encoding are usually called preprocessing, while creating new information, such as a ratio or a rolling average, is feature engineering. In practice both belong inside the same reproducible pipeline. ## Where it came from Constructing derived variables is as old as statistics. Econometricians and biostatisticians have long used logarithms, ratios and interaction terms in regression models, and George Box and David Cox published their family of power transformations in 1964 in the Journal of the Royal Statistical Society, Series B. In-Kwon Yeo and Richard Johnson extended the idea to negative values in Biometrika in 2000. In information retrieval, Karen Spärck Jones introduced inverse document frequency in 1972, the basis of TF-IDF text features. For categorical data, Daniele Micci-Barreca's 2001 paper in SIGKDD Explorations, "A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems", set out smoothed target encoding. Kilian Weinberger and colleagues described feature hashing for large-scale learning at ICML 2009. CatBoost's authors, Liudmila Prokhorenkova and colleagues, introduced ordered target statistics at NeurIPS 2018 to avoid the leakage that naive target encoding causes. For relational data, James Max Kanter and Kalyan Veeramachaneni at MIT presented Deep Feature Synthesis at IEEE DSAA 2015, an automatic way to stack aggregations across linked tables. Deep learning reduced the need for hand-made features in images, audio and text, because networks learn their own representations. On tabular data the picture has been slower to change: Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux's NeurIPS 2022 paper "Why do tree-based models still outperform deep learning on typical tabular data?" found that tree ensembles remained strong. More recently, tabular foundation models such as [TabPFN](/tools/tabpfn), described in Nature in January 2025 by Noah Hollmann and colleagues, learn from many synthetic datasets and need much less manual work on small tables. ## How it works ### Numeric transforms Many real quantities, such as income, prices and counts, are right-skewed: most values are small and a few are huge. A log transform compresses the long tail, so that a model sees "twice as large" as a constant step. Power transforms (Box-Cox for positive data, Yeo-Johnson for any sign) search for the exponent that makes a feature most nearly normal. Ratios often carry more signal than raw values: debt divided by income, clicks divided by impressions, or price per square metre. Differences capture change: today's temperature minus yesterday's. ```text Log transform: x′ = log(1 + x) Cyclic encoding: x_sin = sin(2π · t / P), x_cos = cos(2π · t / P) Smoothed target enc: enc(c) = (n_c · ȳ_c + m · ȳ) / (n_c + m) TF-IDF weight: w(t, d) = tf(t, d) · log(N / df(t)) Interaction: x_ab = x_a · x_b ``` Here x is a raw non-negative value and x′ its transformed version; t is a periodic quantity such as hour of day and P its period (24 for hours, 7 for weekdays); c is a category, n_c the number of training rows with that category, ȳ_c their mean target, ȳ the overall mean target and m a smoothing weight that pulls rare categories towards the overall mean; tf(t, d) is how often term t appears in document d, N is the number of documents and df(t) the number of documents containing t; x_a and x_b are two features whose product forms an interaction. Libraries use slightly different smoothing variants of the IDF term. Tree models are invariant to monotonic transforms of a single feature, so a log transform changes little for a [random forest](/tools/random-forest) or [XGBoost](/tools/xgboost). Ratios and differences still help trees, because a tree would need many splits to approximate them. ### Binning Binning turns a number into a category: age into age bands, distance into "under 1 km, 1 to 5 km, over 5 km". Quantile bins put equal numbers of rows in each band. Binning lets a linear model capture a curved effect and reduces the influence of outliers. It also throws information away, so for flexible models it rarely helps. Splines are a smoother alternative: scikit-learn's `SplineTransformer` creates piecewise polynomial basis features that let a linear model bend smoothly. ### Interactions An interaction captures an effect that depends on two features together. Ice-cream sales may rise with temperature only on weekends. A product term (temperature × weekend) gives a linear model that pattern. `PolynomialFeatures` creates all products up to a degree, but the count explodes with many features, so targeted interactions chosen from domain knowledge are usually better. Tree ensembles and neural networks find interactions on their own, though explicit ratios can still speed them up. ### Date and time features A timestamp hides many features: hour, day of week, day of month, month, quarter, whether it is a public holiday or the day before one, and time since a reference event such as the customer's sign-up or last purchase. Periodic features have a wrap-around problem: hour 23 and hour 0 are neighbours, but as numbers they are far apart. Encoding the hour as a sine and cosine pair places it on a circle and fixes this. For forecasting, lag features (the value one day, one week and one year ago) and rolling statistics (mean of the last seven days) are often the strongest inputs of all, as in classical ARIMA-style thinking. Every such feature must use only information available at prediction time. ### Text features The simplest text features are counts: length, number of words, presence of keywords. Bag-of-words and TF-IDF turn a document into a sparse vector of weighted word counts that linear models handle well. Character n-grams cope with typos and new words. Pre-trained language models produce dense [embeddings](/tools/embeddings) that capture meaning, so that "late delivery" and "parcel arrived after a week" end up close together. For tabular models, embeddings are often reduced with principal component analysis or summarised with a small classifier before being added as features. ### Target encoding and leakage High-cardinality categories, such as postcode or product ID, can have thousands of values. One-hot encoding them creates thousands of sparse columns. Target encoding replaces each category with the average target for that category, smoothed towards the overall mean when the category is rare. It is compact and powerful, and dangerous. If the encoding for a row is computed from data that includes that row's own target, the feature contains a piece of the answer. The model learns to trust it, and validation scores become falsely high, especially for rare categories. The fix is cross-fitting: split the training data into folds and encode each fold using statistics computed from the other folds only. scikit-learn's `TargetEncoder`, added in version 1.3, does this automatically inside `fit_transform`, and its documentation warns that `fit(X, y).transform(X)` is not equivalent and can leak. CatBoost uses ordered target statistics, which encode each row using only the rows before it in a random order. At prediction time, encodings come from the full training data. Target encoding is only one source of leakage. Others include features computed after the outcome (a "refund issued" flag when predicting returns), aggregations over windows that include the future, and statistics computed over the whole dataset before splitting. ### Aggregations for relational data Business data usually lives in several linked tables: customers, orders, order lines, support tickets. To predict something about a customer, you aggregate their related rows: count of orders, total spend, mean and maximum order value, days since last order, number of distinct product categories, and trends such as spend in the last 30 days divided by spend in the previous 90. Deep Feature Synthesis automates this by applying aggregation and transform functions along the table relationships and stacking them to a chosen depth. The key discipline is a cut-off time: for each prediction, only rows that existed before that moment may be aggregated. A [feature store](/tools/feature-store) helps keep these definitions identical in training and serving, and tools such as DuckDB or Polars make window aggregations fast. ### Feature selection More features are not always better. Irrelevant features add noise and variance; redundant ones waste compute and confuse interpretation. Selection methods fall into three groups. - **Filter methods** score each feature on its own, for example by correlation, mutual information or a chi-squared test, and keep the best. They are fast but ignore interactions. - **Wrapper methods** search over subsets using a model's validation score, such as recursive feature elimination or forward selection. They are accurate but expensive and can overfit the validation data. - **Embedded methods** select as part of training: L1 [regularisation](/tools/regularization) sets weights to zero, and tree models give importance scores. Permutation importance, which measures how much the validation score drops when one feature's values are shuffled, is more reliable than split-count importance. Boruta, described by Miron Kursa and Witold Rudnicki in the Journal of Statistical Software in 2010, compares each feature with shuffled "shadow" copies. Selection must happen inside the cross-validation loop. Choosing features on the full dataset and then cross-validating gives optimistic scores. [SHAP](/tools/shap) values help explain which features a model relies on, but explanation is not the same as selection. ### When deep learning and foundation models reduce the need Convolutional networks learn edge and texture detectors that once had to be designed by hand, and transformers learn text representations that replace hand-built n-gram features. On tabular data, models such as FT-Transformer learn embeddings for each column, and tabular foundation models such as TabPFN and [TabICL](/tools/tabicl) are pre-trained to perform prediction in context, often doing well on small tables with minimal preprocessing; see [tabular foundation models](/tools/tabular-foundation-models). They reduce the need for transforms, scaling and interaction terms. They do not remove the need for features that add new information, such as aggregations from other tables, lag features or domain ratios, and they cannot undo leakage. ### A short code example ```python import numpy as np from sklearn.compose import ColumnTransformer from sklearn.preprocessing import FunctionTransformer, SplineTransformer, TargetEncoder from sklearn.pipeline import make_pipeline from sklearn.linear_model import LogisticRegression def hour_cyclic(X): h = np.asarray(X, dtype=float) return np.hstack([np.sin(2 * np.pi * h / 24), np.cos(2 * np.pi * h / 24)]) features = ColumnTransformer([ ("log_amount", FunctionTransformer(np.log1p), ["amount"]), ("hour", FunctionTransformer(hour_cyclic), ["hour"]), ("age_spline", SplineTransformer(n_knots=5), ["age"]), ("merchant_te", TargetEncoder(), ["merchant_id"]), # cross-fitted in fit_transform ]) model = make_pipeline(features, LogisticRegression(max_iter=1000)) model.fit(X_train, y_train) ``` ## Key concepts and terms | Term | Meaning | |---|---| | Feature | An input column the model uses to make predictions | | Derived feature | A new feature computed from one or more raw columns | | Interaction | A feature capturing the joint effect of two or more inputs | | Cyclic encoding | Representing periodic values as sine and cosine so the ends meet | | Lag feature | A past value of a time series used as an input | | Target encoding | Replacing a category with a smoothed average of the target | | Cross-fitting | Computing encodings for each fold from the other folds only | | Feature hashing | Mapping categories or tokens to a fixed number of columns with a hash function | | Cut-off time | The moment before which data may be used for a given prediction | | Leakage | Information in a feature that would not be available at prediction time | | Permutation importance | Drop in validation score when one feature is shuffled | | Feature selection | Choosing a subset of features to keep | ## A worked example This is an illustrative scenario. An online shop wants to predict whether a customer will make another purchase in the next 30 days. The raw data has a customers table (sign-up date, country) and an orders table (customer ID, timestamp, amount, items, returned flag). The team builds a training set of 50,000 customer snapshots, each with a cut-off date and a label of whether a purchase followed within 30 days. **Baseline.** Using only country and days since sign-up, a gradient-boosting model reaches ROC-AUC 0.66. **Recency, frequency and monetary features.** For each snapshot, using only orders before the cut-off: days since last order, number of orders in the last 30, 90 and 365 days, total and mean spend, and share of orders returned. ROC-AUC rises to 0.79. **Trend and time features.** Orders in the last 30 days divided by orders in the previous 90, the weekday of the last order, and whether the cut-off falls in the four weeks before a major holiday. ROC-AUC 0.81. **A leakage scare.** A teammate adds "total lifetime orders" computed over the whole table. ROC-AUC jumps to 0.93. Checking the definition shows it counts orders after the cut-off, including the very purchase being predicted. The feature is rebuilt with the cut-off and ROC-AUC returns to 0.81, the honest number. **Selection.** Permutation importance on a validation fold shows that 9 of 31 features contribute almost nothing. Removing them keeps ROC-AUC at 0.81 and halves feature computation time in production. ## Strengths and limitations **Strengths:** - Often the biggest single lever on tabular model accuracy. - Brings domain knowledge into the model in a transparent way. - Can make simple, explainable models competitive with complex ones. - Good aggregations turn relational and event data into a single learnable table. - Selection reduces cost, latency and the risk of spurious patterns. **Limitations:** - Time-consuming and dependent on domain expertise. - The main source of leakage when done carelessly. - Features must be computed identically in training and production, which adds engineering work. - Hand-made features can encode assumptions or biases that later prove wrong. - Gains shrink for models that learn representations themselves, such as deep networks on images or text. ## How it compares with alternatives | Approach | What it offers | Trade-off | |---|---|---| | Manual feature engineering | Domain-informed, interpretable features | Slow, requires expertise | | Automated feature synthesis | Many aggregation features generated systematically | Can produce thousands of weak features; needs selection | | Representation learning in deep networks | Learns features from raw data | Needs more data; less effective on many tabular tasks | | [Tabular foundation models](/tools/tabular-foundation-models) | Strong results on small tables with little preprocessing | Limits on table size; still needs informative raw columns | | Tree ensembles on raw columns | Handle interactions and monotonic transforms automatically | Still benefit from ratios, lags and aggregations | ## Practical guidance and best practices 1. Start from the prediction moment: for every feature, ask whether its value would truly be known at that time. 2. Build features inside a pipeline so that the same code runs in training, validation and production. 3. Use cross-fitted target encoding, never encodings computed on the full training set including each row's own target. 4. Prefer ratios, differences and recent-window aggregations; they often carry the most signal. 5. Encode periodic quantities, such as hour and month, with sine and cosine pairs for models that are not tree-based. 6. Add features in small groups and measure the change on validation data each time. 7. Do feature selection inside cross-validation, and prefer permutation importance over split counts. 8. Document every feature: definition, source, time window and owner. 9. Monitor feature distributions in production; a broken upstream source often shows up as a sudden shift. ## Common mistakes - Aggregating over all rows, including those after the prediction moment. - Fitting a target encoder on the full training set and then using it on the same rows. - Creating hundreds of polynomial features for a linear model without regularisation. - Encoding hour or month as a plain number for a linear or distance-based model. - Selecting features on the whole dataset before cross-validation. - Computing a feature differently in the training notebook and in the production service. - Trusting a sudden large jump in validation score without checking for leakage. ## Frequently asked questions ### Do tree-based models need feature engineering? Less than linear models, because trees handle monotonic transforms, missing values and many interactions automatically. But trees cannot invent information that is not in the table. Aggregations across related rows, lag features, ratios and domain flags still commonly bring large gains for [gradient boosting](/tools/gradient-boosting) and random forests. ### What is the safest way to use target encoding? Use cross-fitting, so each training row's encoding comes from other folds, add smoothing so rare categories are pulled towards the overall mean, and fit the encoder inside your cross-validation pipeline. scikit-learn's `TargetEncoder` does the cross-fitting in `fit_transform`, and CatBoost does a similar job with ordered target statistics. ### How many features are too many? There is no fixed number. Problems arise when many features are noise relative to the number of training rows, when they are highly redundant, or when they are expensive to compute in production. Watch validation performance and stability across folds as you add features, and remove those whose permutation importance is near zero. ### Should I use embeddings for text columns in a tabular model? Often yes, when the text carries meaning that keywords miss. Compute the embeddings with a pre-trained model, reduce their dimension if the table is small, and treat them as features. Compare against simple TF-IDF features, which are cheap and sometimes just as good for short, repetitive text such as product categories. ### Is automated feature engineering worth it? It is useful for relational data with many tables and standard aggregations, because it explores combinations systematically and quickly. It tends to produce many redundant features, so pair it with selection and with a strict cut-off time to avoid leakage. Domain-specific features usually still need a human. ### Do tabular foundation models remove the need for feature engineering? They remove much of the routine work, such as scaling, transforms and many interaction terms, especially on small datasets. They do not remove the need for features that bring in new information, such as aggregates from other tables, lags or business rules, and they inherit any leakage present in the inputs. ## Glossary - Feature: an input variable used by a model. - Box-Cox transform: a power transform for strictly positive data that makes it closer to normal. - Yeo-Johnson transform: a power transform that also works with zero and negative values. - Spline features: piecewise polynomial basis functions that let linear models bend smoothly. - TF-IDF: term frequency weighted by inverse document frequency. - Rolling window: a statistic computed over a fixed recent period. - Deep Feature Synthesis: automatic stacking of aggregations and transforms across related tables. - Filter method: feature selection by scoring each feature on its own. - Wrapper method: feature selection by searching subsets with a model. - Embedded method: feature selection that happens during model training. ## Summary Feature engineering turns raw data into inputs that make patterns easy to learn: transforms for skewed numbers, bins and splines for curved effects, interactions for joint effects, calendar, lag and cyclic features for time, counts, TF-IDF and embeddings for text, encodings for categories, and aggregations across related tables for relational data. Selection then removes what does not help. The biggest danger is leakage, especially from naive target encoding and from aggregations that see the future, so every feature must respect the prediction moment and be computed inside a cross-validated pipeline. Deep learning and tabular foundation models reduce the routine work, but features that add genuinely new information remain one of the strongest levers in tabular machine learning.