# Linear regression: your points, gradient descent and the loss surface

The simplest model that learns from data, drawn completely. Place your own points, then watch gradient descent walk down the loss surface to the same answer the normal equation gives in one step.

<div data-sim="linreg"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The left plot is your data with the current model (glowing) and the exact least-squares fit (dashed). The upper right panel is the loss over every possible slope and intercept, with the path gradient descent takes; the circle marks the optimum.

## Things to try

- Drag across the plot to add many points, then train.
- Raise the learning rate until the path zig-zags across the valley and diverges.
- Add one far-away point and compare MSE with Huber loss.
- Switch to the cubic and fit a curve with the same linear machinery.

## How it works in real systems

Linear models are still the first thing to try on tabular data: fast, interpretable and hard to overfit. scikit-learn's LinearRegression solves the normal equation (via a least-squares solver); SGDRegressor uses gradient descent for data that does not fit in memory.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Data and the closed-form answer

For mean squared error, the best line has an exact formula: the normal equation (XᵀX)θ = Xᵀy.

```python
import numpy as np
rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, 50); y = 0.8 * x - 0.2 + rng.normal(0, 0.1, 50)
X = np.c_[np.ones_like(x), x]                 # a column of ones for the intercept
theta = np.linalg.solve(X.T @ X, X.T @ y)     # solve, never invert explicitly
print("intercept, slope:", theta)
```

### Step 2 · The same with gradient descent

The gradient of MSE with respect to θ is (2/n)·Xᵀ(Xθ − y). Step against it.

```python
theta_gd, lr = np.zeros(2), 0.3
for step in range(500):
    grad = 2 / len(y) * X.T @ (X @ theta_gd - y)
    theta_gd -= lr * grad
print(theta_gd)   # matches the closed form to ~6 decimals
```

Gradient descent is pointless for two parameters, but it is the only option for a neural network with millions, which has no closed form.

### Step 3 · Robust losses and more features

Huber loss behaves like MSE for small errors and like MAE for large ones, so one wild point does not drag the line. Polynomial features make a curve from a linear model.

```python
from sklearn.linear_model import LinearRegression, HuberRegressor
from sklearn.preprocessing import PolynomialFeatures
y_out = y.copy(); y_out[0] += 3                             # one outlier
print(LinearRegression().fit(x[:, None], y_out).coef_)       # pulled by the outlier
print(HuberRegressor().fit(x[:, None], y_out).coef_)         # barely moves

X3 = PolynomialFeatures(3, include_bias=False).fit_transform(x[:, None])
m = LinearRegression().fit(X3, y)
print("R²", m.score(X3, y))
```

### Step 4 · Evaluate honestly

R² on the training data always rises with more features. Report it on held-out data.

```python
from sklearn.model_selection import cross_val_score
print(cross_val_score(LinearRegression(), x[:, None], y, cv=5, scoring="r2").mean())
```

<div class="cl-end"></div>

## More simulations

[← LangGraph state](/ai-simulations/langgraph-state-checkpoints) · [All AI Simulations](/ai-simulations) · [Decision tree →](/ai-simulations/decision-tree)
