Linear regression: your points, gradient descent and the loss surface
The simplest model that learns from data, drawn completely. Place your own points, then watch gradient descent walk down the loss surface to the same answer the normal equation gives in one step.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The left plot is your data with the current model (glowing) and the exact least-squares fit (dashed). The upper right panel is the loss over every possible slope and intercept, with the path gradient descent takes; the circle marks the optimum.
Things to try
- Drag across the plot to add many points, then train.
- Raise the learning rate until the path zig-zags across the valley and diverges.
- Add one far-away point and compare MSE with Huber loss.
- Switch to the cubic and fit a curve with the same linear machinery.
How it works in real systems
Linear models are still the first thing to try on tabular data: fast, interpretable and hard to overfit. scikit-learn's LinearRegression solves the normal equation (via a least-squares solver); SGDRegressor uses gradient descent for data that does not fit in memory.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Data and the closed-form answer
For mean squared error, the best line has an exact formula: the normal equation (XᵀX)θ = Xᵀy.
import numpy as np
rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, 50); y = 0.8 * x - 0.2 + rng.normal(0, 0.1, 50)
X = np.c_[np.ones_like(x), x] # a column of ones for the intercept
theta = np.linalg.solve(X.T @ X, X.T @ y) # solve, never invert explicitly
print("intercept, slope:", theta)
Step 2 · The same with gradient descent
The gradient of MSE with respect to θ is (2/n)·Xᵀ(Xθ − y). Step against it.
theta_gd, lr = np.zeros(2), 0.3
for step in range(500):
grad = 2 / len(y) * X.T @ (X @ theta_gd - y)
theta_gd -= lr * grad
print(theta_gd) # matches the closed form to ~6 decimals
Gradient descent is pointless for two parameters, but it is the only option for a neural network with millions, which has no closed form.
Step 3 · Robust losses and more features
Huber loss behaves like MSE for small errors and like MAE for large ones, so one wild point does not drag the line. Polynomial features make a curve from a linear model.
from sklearn.linear_model import LinearRegression, HuberRegressor
from sklearn.preprocessing import PolynomialFeatures
y_out = y.copy(); y_out[0] += 3 # one outlier
print(LinearRegression().fit(x[:, None], y_out).coef_) # pulled by the outlier
print(HuberRegressor().fit(x[:, None], y_out).coef_) # barely moves
X3 = PolynomialFeatures(3, include_bias=False).fit_transform(x[:, None])
m = LinearRegression().fit(X3, y)
print("R²", m.score(X3, y))
Step 4 · Evaluate honestly
R² on the training data always rises with more features. Report it on held-out data.
from sklearn.model_selection import cross_val_score
print(cross_val_score(LinearRegression(), x[:, None], y, cv=5, scoring="r2").mean())