Linear regression

Overview

Linear regression predicts a number, such as a house price, a delivery time or next month's sales, as a weighted sum of input features plus a constant. Each feature gets one weight, called a coefficient. The model multiplies each feature by its weight, adds everything up, and adds a starting value called the intercept. Fitting the model means choosing the weights that make the predictions as close as possible to the known answers in the training data.

It is the simplest predictive model that is still genuinely useful. It trains in a fraction of a second, it can be written on one line, and every weight has a plain meaning. That is why it is usually the first model to try and the baseline that more complex models must beat.

An everyday picture: when you price a flat by rule of thumb, you start with a base price, add so much per square metre, add a premium for a balcony and subtract something for every kilometre from the centre. That mental formula is a linear regression. The only difference is that the computer finds the numbers from data instead of from habit.

What it is

The model is written as y = b0 + b1·x1 + b2·x2 + ... + bp·xp. "Linear" means linear in the weights, not necessarily in the raw inputs. You can add squared terms, logarithms or interactions as extra features and the model is still a linear regression. This is how a straight-line method can describe curved relationships.

There are two common reasons to use it. The first is prediction: get a good number for a new case. The second is explanation: understand how strongly each factor is associated with the outcome. Statistics courses emphasise the second; machine learning work often emphasises the first. The same fitted model serves both, but explanation needs more care with assumptions.

Where it came from

The method of least squares, which is how linear regression is usually fitted, was first published clearly by Adrien-Marie Legendre in 1805, in a work on determining the orbits of comets. Carl Friedrich Gauss published his own treatment in 1809 and said he had been using the method since 1795, which led to a well-known priority dispute. Gauss's least-squares calculations had already helped astronomers find the asteroid Ceres again in 1801 after it was lost in the Sun's glare. Gauss also connected the method to the normal distribution of errors, and his later work is the basis of what is now called the Gauss-Markov theorem.

The word "regression" came later, from Francis Galton's studies of inheritance in the nineteenth century. He noticed that sons of very tall fathers tended to be tall, but on average less extreme than their fathers, and he called this regression towards the mean. The name stuck to the whole family of methods.

In the twentieth century regression became the workhorse of economics, social science, medicine and engineering. Regularised versions followed: ridge regression in the 1970s and the lasso in the 1990s, both designed to keep estimates stable when there are many or correlated features. Today linear regression is built into every statistics package and machine learning library, including scikit-learn.

How it works

Step 1: set up the data

Put the training data in a table. Each row is one example, each column is a feature, and one column is the target. Convert categories to numbers, usually with one-hot encoding, and decide how to handle missing values.

Step 2: choose the loss

Ordinary least squares (OLS) chooses the weights that minimise the sum of squared residuals. Squaring makes all errors positive and punishes large errors more than small ones. It also gives a smooth bowl-shaped loss surface with a single lowest point.

Step 3: solve for the weights

For OLS there is a closed-form solution from linear algebra, often called the normal equations. In practice libraries use numerically stable methods such as QR decomposition or singular value decomposition. For very large data, the weights can be found by gradient descent, which takes small steps downhill on the loss surface; see optimization.

Step 4: add regularisation if needed

When features are many or strongly correlated, plain OLS weights can swing wildly. Ridge regression adds a penalty on the sum of squared weights, which shrinks them smoothly. Lasso adds a penalty on the sum of absolute weights, which can push some weights exactly to zero and so selects features. Elastic net mixes the two. The penalty strength is a hyperparameter chosen by cross-validation.

Step 5: check the fit

Look at residuals, not just a single score. Plot residuals against predictions: a random cloud is good, a curve suggests a missing non-linear term, and a funnel shape suggests the error grows with the size of the target. Measure error on held-out data with metrics such as mean absolute error (MAE), root mean squared error (RMSE) and R².

Step 6: interpret carefully

A coefficient says how much the prediction changes per unit of a feature when the other features stay fixed. It does not by itself prove cause and effect. For causal questions see causal inference.

from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
model.fit(X_train, y_train)
print(model.score(X_test, y_test))  # R squared on test data

Key concepts and terms

A worked example

This scenario is illustrative. A small letting agency wants to estimate monthly rent for flats in one city. It has 1,200 past lettings with four features: floor area in square metres, number of bedrooms, distance to the city centre in kilometres, and whether the flat has a balcony.

The team holds out 20 per cent of rows as a test set. On the remaining 960 rows it fits a ridge regression after scaling the features, choosing the penalty with five-fold cross-validation. Converted back to original units, the model reads roughly:

rent = 310 + 14.5 × area + 40 × bedrooms − 22 × distance + 65 × balcony

So a 70 m² two-bedroom flat, 3 km from the centre, with a balcony, is predicted at 310 + 1,015 + 80 − 66 + 65 = 1,404 per month.

On the test set the mean absolute error is 118 per month and R² is 0.78. A residual plot shows the model under-predicts the most expensive flats, so the team tries predicting the logarithm of rent instead. That turns the weights into approximate percentage effects and reduces the error on large flats. The final model is used as a quick quote tool, and as the baseline against which a gradient boosting model is later compared.

Strengths and limitations

Strengths:

• Very fast to train and to run, even on millions of rows.
• Each weight has a direct, readable meaning.
• Few hyperparameters; regularised versions are hard to break.
• Works well with small datasets where complex models overfit.
• Well-understood statistical theory for confidence intervals and tests.

Limitations:

• Captures only additive, linear effects unless you engineer features by hand.
• Sensitive to outliers because errors are squared.
• Weights become unstable when features are highly correlated.
• Extrapolation outside the training range can be badly wrong.
• Interpretation as cause and effect is often unjustified.

How it compares with alternatives

Practical guidance and best practices

1. Start every regression project with a linear baseline before trying complex models.
2. Scale features before using ridge or lasso, so the penalty treats them fairly.
3. Plot residuals against predictions and against each feature.
4. Transform skewed targets, for example with a logarithm, when errors grow with size.
5. Use cross-validation to choose the regularisation strength.
6. Check correlation between features and drop or combine near-duplicates.
7. Report error in the target's own units, such as MAE in euros, not only R².
8. Keep the preprocessing and model in one pipeline so the same steps run in production.

Common mistakes

• Reading coefficients as causal effects without a causal design.
• Comparing raw coefficients across features measured in different units.
• Fitting scalers or encoders on the full dataset before splitting, which leaks information.
• Ignoring outliers that dominate a squared-error fit.
• Trusting predictions far outside the range of the training data.
• Judging the model by training R² alone.

Frequently asked questions

Is linear regression machine learning or statistics?

It is both. Statisticians use it mainly to estimate and test relationships, with attention to assumptions and uncertainty. Machine learning practitioners use the same model mainly to predict new values and judge it by held-out error. The mathematics is identical; the emphasis differs.

Can linear regression model curves?

Yes, if you give it curved features. Adding a squared term, a logarithm or a spline basis lets the model fit a bend while staying linear in its weights. The limit is that you must choose those features yourself, whereas tree-based models discover such shapes automatically.

When should I use ridge instead of lasso?

Use ridge when you expect many features to matter a little and some are correlated; it shrinks them together. Use lasso when you suspect only a few features matter and want the model to drop the rest. If unsure, elastic net combines both and cross-validation can pick the mix.

What does R² actually tell me?

R² is the share of variation in the target that the model explains compared with always predicting the mean. A value of 0.78 means the model explains 78 per cent of that variation. It does not tell you whether errors are acceptable in practice, so always report an error in real units as well.

Do the features need to be normally distributed?

No. The classical assumption concerns the residuals, not the features, and it matters mainly for confidence intervals and hypothesis tests. For pure prediction, what matters most is that the relationship is roughly linear in the chosen features and that the test data resemble the training data.

Why is it still used when better models exist?

Because it is fast, stable, transparent and hard to overfit badly. On small or noisy datasets it often performs as well as complex models, and in regulated settings its readability can outweigh a small gain in accuracy.

Glossary

• Feature: an input variable used to make a prediction.
• Target: the numeric value the model predicts.
• Coefficient: the learned weight for one feature.
• Residual: the difference between the true and predicted value.
• Least squares: fitting by minimising the sum of squared residuals.
• Regularisation: a penalty that discourages large weights.
• R²: the proportion of target variance the model explains.
• Multicollinearity: strong correlation among features.
• One-hot encoding: turning a category into a set of 0 or 1 columns.

Summary

Linear regression predicts a number as a weighted sum of features plus an intercept, with weights chosen by least squares, a method published by Legendre in 1805 and Gauss in 1809. It is fast, transparent and a strong baseline, and regularised forms such as ridge and lasso keep it stable with many features. Its limits are linear, additive effects, sensitivity to outliers and the temptation to read correlation as causation. Use it first, check residuals, validate on held-out data, and move to trees or boosting only when the data show they add real value.

Related

It belongs to ML foundations in my stack, under Machine learning. Others in the same group: