Özer Cem KELAHMETKELAHMET
AI Simulations · Large tabular foundation model

In-context regression: predicting numbers from the rows of the table

The same in-context mechanism for numbers. The query row attends to the labelled rows and reads off a price with an interval, and bends where a straight-line model cannot.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The plot is a slice through the selected query row along one column: the in-context prediction with its 10–90% band, ridge regression (dashed), and the context rows sized by attention. The table shows each query's prediction with its uncertainty, and the curve shows error on 300 unseen rows as the context grows.

Things to try

  • Slice along distance: the city-centre premium is a curve, not a line.
  • Use only 8 context rows and look at the band widen.
  • Edit a price in the context and watch the nearby predictions follow.

How it works in real systems

Regression with a tabular foundation model works the same way as classification; models such as TabPFN predict a full distribution over the target, from which the mean, the median and any interval follow.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Make the table

import numpy as np
rng = np.random.default_rng(11)
def houses(n):
    s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
    p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
    return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)

Step 2 · In-context regression from scratch

def icl_reg(Xc, yc, Xq, tau=0.4):
    mu, sd = Xc.mean(0), Xc.std(0) + 1e-9
    Zc, Zq = (Xc - mu) / sd, (Xq - mu) / sd
    w = np.abs([np.corrcoef(Zc[:, f], yc)[0, 1] for f in range(Xc.shape[1])]); w = .25 + .75 * w / w.mean()
    d = (((Zq[:, None] - Zc[None]) ** 2) * w).sum(-1) / w.sum()
    a = np.exp(-(d - d.min(1, keepdims=True)) / tau); a /= a.sum(1, keepdims=True)
    mean = a @ yc
    sdv = np.sqrt((a * (yc[None] - mean[:, None]) ** 2).sum(1))
    return mean, sdv
m, s = icl_reg(X, y, X_test)
print("RMSE", np.sqrt(((m - y_test) ** 2).mean()))

Step 3 · TabPFN for regression

from tabpfn import TabPFNRegressor
from sklearn.linear_model import Ridge
from sklearn.metrics import root_mean_squared_error
reg = TabPFNRegressor().fit(X, y)
print("TabPFN", root_mean_squared_error(y_test, reg.predict(X_test)))
print("ridge ", root_mean_squared_error(y_test, Ridge(1.0).fit(X, y).predict(X_test)))

The price has a city-centre premium that fades with distance; a linear model cannot bend, the in-context model can.

More simulations

← In-context classification · All AI Simulations · Probabilistic regression →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open