Linear algebra

Overview

Linear algebra is the mathematics of vectors, matrices and the operations that connect them. It is the working language of nearly all modern machine learning. A table of numbers is a matrix. A row is a vector. A layer of a neural network is a matrix multiplication followed by a simple nonlinearity. Once you see data this way, a huge range of methods become variations on a few simple moves.

For tabular data, linear algebra is the most natural lens there is. A dataset with a thousand rows and twenty columns is a thousand-by-twenty matrix. Cleaning, scaling, compressing, comparing and predicting from it all reduce to matrix operations that run fast on a GPU. Tabular foundation models lean on exactly these operations, especially in their attention layers and embeddings. See /model/architecture for how this appears in a model, and Optimization for how the matrices are learned.

What it is

Linear algebra studies linear structure: things you can add together and scale. Its main objects are listed below.

• Scalars are single numbers.
• Vectors are ordered lists of numbers, such as one row of a table.
• Matrices are grids of numbers, such as a whole table or a layer's weights.
• Tensors are higher-dimensional grids, such as a batch of tables.

Its main operations are adding, scaling, multiplying, transposing, inverting and decomposing these objects.

Why "linear"

A function is linear if doubling the input doubles the output, and the output for a sum of inputs is the sum of the outputs. Linear functions are simple, predictable and fast. Real patterns are rarely linear, so neural networks stack many linear steps with nonlinear steps in between. The linear parts do the heavy lifting in terms of computation.

A table as a matrix

Take a table with n rows and d numeric columns. Write it as a matrix X of shape n by d. Each row x_i is a point in a d-dimensional space. Columns are the axes of that space. Two rows that are similar sit close together. A model is, in this picture, a rule for drawing useful boundaries or surfaces through that cloud of points.

An everyday analogy: a spreadsheet is a matrix. Sorting and filtering are about rows. Summing a column is a vector operation. A pivot table is a structured matrix product in disguise.

Where it came from

The roots are old. Systems of linear equations were solved by hand in ancient times, and methods for eliminating variables are among the oldest algorithms known. The idea of determinants and the formal study of matrices developed in the seventeenth to nineteenth centuries, and the abstract theory of vector spaces took shape in the nineteenth and early twentieth centuries.

Numerical linear algebra, the art of computing with matrices on finite-precision machines, grew with early computers. Decompositions that are now standard, such as the QR decomposition, the singular value decomposition and the eigenvalue algorithms, were turned into stable and fast routines in the mid-twentieth century. Libraries such as BLAS and LAPACK standardized the basic building blocks, and almost every modern numerical package still sits on top of them.

Deep learning added a new chapter. GPUs, built for graphics, turned out to be superb at dense matrix multiplication. Frameworks such as PyTorch and JAX made it easy to express models as chains of matrix operations and to differentiate through them. The "attention is all you need" paper of 2017 introduced the Transformer, whose core is a pair of matrix products. Since then, much of the progress in model speed has come from smarter ways to schedule and fuse matrix operations, as in FlashAttention and mixed precision training.

How it works

This section builds the main ideas in order. Each one is the bridge to the next.

Step 1: Vectors and what you can do with them

A vector can be added to another vector, scaled by a number, and compared to another vector. The most important comparison is the dot product: multiply matching entries and add them up. The dot product is large when two vectors point in the same direction, zero when they are perpendicular, and negative when they oppose each other.

Analogy: two people walking. If they walk in the same direction, their dot product is large and positive. If one walks north and the other east, it is zero. The dot product measures how much one person's walk helps the other along.

Step 2: Length and distance

The norm of a vector is its length. The straight-line distance between two rows is the norm of their difference. Many methods, including nearest-neighbor search and clustering, use these distances. Cosine similarity divides the dot product by both lengths, so it compares direction and ignores size.

Step 3: Matrices as machines that transform vectors

A matrix A acts on a vector v to produce a new vector Av. It can rotate, stretch, squash, reflect or project. A neural network layer is this move plus a bias and a nonlinearity. When A has fewer rows than columns, it maps to a lower-dimensional space and compresses. When it has more, it expands.

Step 4: Matrix multiplication

Multiplying two matrices composes their transformations. If A is n by k and B is k by m, then AB is n by m. Each entry is a dot product between a row of A and a column of B. This one operation is where nearly all of the compute in deep learning is spent.

Analogy: a recipe scaled for many dinners at once. Each row of A is a recipe, each column of B is a shopping list for one dinner, and each entry of the product is the cost of one recipe for one dinner.

Step 5: Solving linear systems and least squares

Given Ax = b, find x. If there are more equations than unknowns, as in a typical table with many rows and few columns, there is usually no exact solution, so you find the x that makes the squared error smallest. This is least squares, and it is the engine of linear regression. The solution can be written with the normal equations, but in practice it is computed using stable decompositions rather than by explicitly inverting a matrix.

Step 6: Eigenvectors and eigenvalues

Some special vectors are only stretched, not turned, by a matrix. These are eigenvectors, and the stretch factors are eigenvalues. For a covariance matrix of columns, the eigenvectors point along the main directions of variation, and the eigenvalues say how much variation lies along each. This is the idea behind principal component analysis.

Step 7: The singular value decomposition

The SVD writes any matrix as a product of three simple pieces: a rotation, a set of stretches along axes, and another rotation. The stretches are the singular values. Keeping only the largest few gives the best low-rank approximation of the matrix. This is how you compress a table, remove noise, or fit a low-dimensional model of its structure.

Step 8: Rank and redundancy

The rank of a matrix is the number of independent directions it contains. If a column is an exact sum of two other columns, it adds no new rank. Near-redundant columns make some methods unstable, because tiny changes in the data cause big changes in the fit. This is called ill-conditioning.

Step 9: Putting it together in a network

A feed-forward layer computes h = f(Wx + b). An attention layer computes similarity scores between all pairs of items with a matrix product, turns the scores into weights, and then takes weighted sums of value vectors with another matrix product. Everything else is bookkeeping around these products.

Four ways to picture a matrix

A single matrix can be read in several complementary ways. Switching between them is a useful skill.

• As a table. Rows are cases and columns are measurements.
• As a transformation. It takes vectors in and gives vectors out.
• As a set of columns. Each column is a direction, and the matrix mixes them.
• As a network of weights. Entry (i, j) is the strength of the link from input j to output i.

Special matrices worth recognizing

• Diagonal matrices only scale each axis. They are cheap to store and to multiply.
• Symmetric matrices equal their own transpose. Covariance matrices are symmetric and have real eigenvalues.
• Orthogonal matrices rotate or reflect without changing lengths. They are the most stable transformations.
• Triangular matrices have zeros on one side of the diagonal. Systems with them are solved by simple back-substitution.
• Sparse matrices are mostly zeros. Storing only the non-zero entries saves memory and time.
• Low-rank matrices can be written as a product of two thin matrices.
• Permutation matrices reorder rows or columns. They show why shuffling rows changes nothing for a symmetric method.

Key concepts and terms

Shapes are the grammar

Most bugs in numerical code are shape bugs. The rule for matrix multiplication is that the inner sizes must match. Learning to read shapes aloud, for example "batch by rows by columns times columns by hidden", saves hours.

A worked example

We will use a tiny table to show centering, covariance and a projection onto the main direction, and then show the same steps in code.

The data

Four rows, two columns, say two sensor readings:

The two columns rise together. The column means are 5 and 10. Subtract them to center the data: the centered rows are (-3, -6), (-1, -2), (1, 1) and (3, 7).

Covariance

The covariance matrix is the centered matrix, transposed, times itself, divided by the number of rows minus one. Here the sums are 20 for a squared, 46 for the cross term and 90 for b squared, so the covariance is about [[6.67, 15.33], [15.33, 30.0]].

The largest eigenvector of this matrix points almost along the diagonal direction, where most of the variation lies. Projecting each row onto it gives a single number per row that keeps nearly all of the information in two columns.

In code

import numpy as np

X = np.array([[2, 4], [4, 8], [6, 11], [8, 17]], dtype=float)
Xc = X - X.mean(axis=0)

cov = Xc.T @ Xc / (len(X) - 1)
vals, vecs = np.linalg.eigh(cov)      # ascending order
top = vecs[:, -1]
scores = Xc @ top                     # one number per row

print("eigenvalues", vals.round(3))
print("share of variance", (vals[-1] / vals.sum()).round(4))
print("projection", scores.round(3))

# The same result through the SVD
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
print("singular values", S.round(3))
print("first direction", Vt[0].round(3))

The first eigenvalue carries almost all of the variance. That tells you the two columns are close to redundant, and that one derived column would do almost as well as both.

A least squares fit

import numpy as np

rng = np.random.default_rng(0)
A = rng.normal(size=(100, 3))
true_w = np.array([1.0, -2.0, 0.5])
y = A @ true_w + rng.normal(scale=0.1, size=100)

w, *_ = np.linalg.lstsq(A, y, rcond=None)
print(w.round(3))

Here lstsq solves the least squares problem through a stable decomposition instead of forming an inverse. The recovered weights are close to the true ones.

Cosine similarity between rows

import numpy as np

a = np.array([1.0, 2.0, 3.0])
b = np.array([2.0, 4.0, 6.5])
c = np.array([3.0, -1.0, 0.0])

def cos(u, v):
    return float(u @ v / (np.linalg.norm(u) * np.linalg.norm(v)))

print(round(cos(a, b), 4))   # close to 1: almost the same direction
print(round(cos(a, c), 4))   # near 0: unrelated direction

Row b is nearly a scaled copy of a, so their cosine is close to one even though their lengths differ. This is why cosine similarity is popular for comparing embeddings.

Low-rank reconstruction

import numpy as np

rng = np.random.default_rng(2)
A = rng.normal(size=(50, 3)) @ rng.normal(size=(3, 12))   # rank 3
A_noisy = A + 0.05 * rng.normal(size=A.shape)

U, S, Vt = np.linalg.svd(A_noisy, full_matrices=False)
k = 3
A_k = (U[:, :k] * S[:k]) @ Vt[:k]
err = np.linalg.norm(A - A_k) / np.linalg.norm(A)
print(S.round(2)[:6], round(float(err), 4))

The first three singular values stand far above the rest, and keeping only three directions removes most of the noise. This is the core of matrix completion and denoising.

A single attention step by hand

import numpy as np

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

rng = np.random.default_rng(1)
n, d = 5, 4                       # five items, four features each
X = rng.normal(size=(n, d))
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))

Q, K, V = X @ Wq, X @ Wk, X @ Wv
scores = Q @ K.T / np.sqrt(d)     # (n, n) similarity of every pair
weights = softmax(scores)         # each row sums to one
out = weights @ V                 # weighted mix of value vectors
print(out.shape)

Three matrix products and one normalization are the whole mechanism. Everything in a large model is a repetition and refinement of steps like these.

Reading a model through its matrices

When you open a trained network, each weight matrix has a story.

Singular values of these matrices often reveal that they are close to low rank, which is why compression works.

Numerical precision in one page

Computers store numbers with finite precision. Adding a tiny number to a huge one can lose the tiny one entirely. Subtracting two nearly equal numbers can wipe out most of the useful digits. In matrix work, these effects show up as unstable solutions and loss spikes.

• Half precision has a narrow range and few digits, so it needs loss scaling or a wider exponent format.
• Single precision is the default for most training.
• Double precision is for delicate steps such as solving ill-conditioned systems.
• Accumulating sums in higher precision than the stored values is a common safe pattern.

Strengths and limitations

Strengths

• Fast. Dense matrix products are among the most optimized operations in computing, on both CPUs and GPUs.
• Composable. Complex pipelines become chains of simple, well-understood steps.
• Differentiable. Matrix operations have clean derivatives, so gradient-based training works smoothly.
• Compressible. Low-rank structure lets you shrink models and data with little loss.
• Geometric. Distances, angles and projections give intuition for what a method is doing.
• Mature tooling. Decades of numerical libraries deliver stable, tested routines.

Limitations

• Linear only on its own. Real relationships are nonlinear, so linear algebra needs nonlinear partners.
• Memory hungry. A dense matrix of n by n grows quadratically. Attention over long inputs hits this wall.
• Numerical issues. Ill-conditioned matrices, huge ranges and low-precision arithmetic can produce errors.
• Scale sensitivity. Methods like PCA depend on how columns are scaled, so careless scaling changes the answer.
• Dense by default. Tables with many categorical levels become wide and sparse, which wastes dense compute unless handled carefully.

When a linear model is enough

A linear model on well-chosen features is often a strong baseline on tables. It trains in milliseconds, has no tuning to speak of beyond one penalty, and its weights can be read directly. If a more complex model cannot beat it by a clear margin, the extra complexity is not earning its keep.

How it compares with alternatives

Linear algebra is a toolbox, not a rival method, but it is useful to compare the matrix view with other ways of thinking about the same tasks.

In practice, the strongest systems use both: linear algebra for the heavy arithmetic, and a nonlinear architecture to decide what to compute.

Role in tabular foundation models

Tabular foundation models take raw tables, with no hand-built features, and learn to predict, impute and detect anomalies from them. Linear algebra is how a table becomes something a network can process, and how the network mixes information across rows and columns. Compare /model/architecture and /model/research.

From cells to vectors

Each cell in a table is turned into a vector, an embedding. A numeric cell is often scaled and passed through a small linear map, so a single number becomes a vector of, say, a few hundred values. A categorical cell is looked up in an embedding table, which is a matrix whose rows are category vectors. This is the same as multiplying a one-hot vector by that matrix. The result is that every cell, whatever its type, lives in the same vector space and can be compared with any other.

Mixed types

Mixed types are handled by giving each type its own entry path into the shared space: a linear map for numbers, a lookup for categories, a special learned vector for a missing value. After that, the rest of the network sees only vectors. This is linear algebra acting as a common currency.

No column names

When a table has no usable headers, a model cannot rely on text labels. Instead, it can learn a vector for each column from the data in that column, such as summary statistics of its distribution passed through a linear layer. Two columns with similar distributions end up with similar vectors. Where names are present, a text embedding of the name can be added to the same vector. See Self-supervised learning.

Attention across rows and across columns

A table has two axes, and a tabular network typically runs attention along both. Attention over columns lets a cell look at the other cells in its row, which captures interactions between features. Attention over rows lets a query row compare itself with labeled rows in the context, which is the mechanism behind in-context learning. Each is a matrix product of the form queries times keys, a normalization, and a product with values. See Transformers and TabPFN.

Row order should not matter, because rows in a table are exchangeable. Attention without positional encodings over rows has this symmetry built in, which is a neat linear-algebra fact: the operation is permutation-equivariant.

The cost of the matrix

Attention over n rows creates an n by n score matrix. Memory grows with the square of the row count, so large tables need tricks: attending to a subset of rows, chunking, low-rank approximations and fused kernels as in FlashAttention. Lower precision, as in mixed precision, halves memory and speeds up multiplies. Mixing experts, as in Mixture of experts, splits one large multiply into several smaller ones.

Pre-training on synthetic tables

Synthetic tables used for pre-training are often generated by random processes whose building blocks are themselves linear algebra: random linear maps, random low-rank structure, random mixing of features followed by nonlinear squashing. The network sees many such tables, and learns how to read the structure they hold. See Synthetic data generation and Few-shot learning.

Imputation

Filling in a missing cell can be posed as low-rank matrix completion: assume the table is approximately low rank, then find the low-rank matrix that fits the known entries. Classical methods do this with iterated SVD. Neural imputers generalize the idea, learning nonlinear structure while still using matrix products inside. See Imputation networks.

Anomaly detection

A row that cannot be well reconstructed from the main directions of variation is suspicious. The reconstruction error after projecting onto the top singular directions is a simple, fast anomaly score. Deep versions use autoencoders, but the geometry is the same. See Anomaly detection.

Sector and semantic classification

When a model must label a table or a column by its meaning, such as which industry a dataset belongs to, it compares embedding vectors. Column and table embeddings are compared with each other or with class prototypes using dot products and cosine similarity. The better the geometry of the embedding space, the more cleanly meaning separates.

Continual learning

Updating a model without forgetting old knowledge can be framed in terms of subspaces. Updates that live in directions orthogonal to what old tasks use interfere less. Low-rank adaptation, as in LoRA, adds a small low-rank matrix on top of frozen weights, which is cheap to store and to swap. These are direct uses of rank and orthogonality.

Practical guidance and best practices

Working with matrices in code

• Prefer vectorized operations over Python loops. One matrix product beats a thousand small ones.
• Check shapes at every step, and write them in comments.
• Use solve or lstsq instead of explicitly inverting a matrix.
• Center and scale columns before PCA or any distance-based method.
• Use float32 for most training and float64 for delicate numerical work.
• Keep an eye on the condition number when solving systems.
• Use sparse formats for wide one-hot data.

Choosing a decomposition

Picking a rank

Plot the singular values from largest to smallest. If they fall sharply and then flatten, the knee suggests the rank of the signal. Keep enough components to retain the share of variance you need, for example most of it, and validate on held-out data that the compression does not hurt the task.

Regularization in matrix form

Adding a multiple of the identity matrix to the matrix you invert is the same as ridge regularization. It shrinks weights, stabilizes the solution when columns are nearly redundant and often improves test error. See Statistics for the bias and variance reasoning.

What to monitor in a matrix-heavy training run

Speed tips

• Batch many small operations into one larger multiplication.
• Match matrix sizes to hardware-friendly multiples.
• Use lower precision where training stays stable.
• Avoid copying data between devices more than needed. See CUDA.
• Use fused attention kernels for long inputs.

Linear algebra for feature work

• Interactions. Multiplying two columns adds an interaction term. A linear model with such terms can represent curved surfaces.
• Polynomial features. Squares and cross terms expand the matrix width quickly.
• Target encoding. Replacing a category by a statistic of the target is a linear map applied to a one-hot matrix, and it needs care to avoid leakage.
• Whitening. Rotating and scaling features so their covariance becomes the identity can speed up optimization.
• Random projections. Multiplying by a random matrix preserves distances approximately and reduces width cheaply.

A tiny debugging routine

def check(name, arr):
    print(name, arr.shape, arr.dtype,
          "min", float(arr.min()), "max", float(arr.max()))

# call check on inputs, weights and outputs of each stage

Printing shape, type and range at each stage catches most matrix bugs within minutes.

Common mistakes

1. Shape mismatches hidden by broadcasting. A silent broadcast can turn a vector into a full matrix and give wrong answers without an error.
2. Forgetting to center. PCA on uncentered data finds the mean, not the variation.
3. Scaling after the split in the wrong way. Fit scalers on the training rows only.
4. Inverting matrices explicitly. It is slower and less accurate than solving the system.
5. Ignoring the condition number. A tiny change in data can swing a fit when columns are almost redundant.
6. Mixing up rows and columns. Whether samples are rows or columns changes which covariance you compute.
7. Comparing vectors of different scales. Dot products depend on length. Normalize when you want direction only.
8. Using too many components. Keeping every component keeps the noise.
9. Treating one-hot columns as independent numbers. Their sum is fixed, which creates redundancy when you also include an intercept.
10. Running out of memory on pairwise matrices. An n by n matrix for large n is a common cause of crashes.
11. Overflow in softmax. Subtract the maximum before exponentiating.
12. Assuming linear methods capture interactions. A linear map cannot represent an exclusive-or pattern.

Frequently asked questions

Why is linear algebra so important for machine learning?

Because data and models are naturally expressed as vectors and matrices. A dataset is a matrix, a neural network layer is a matrix multiplication, and attention is a pair of matrix products. Hardware is built to run these operations extremely fast, so expressing a method in matrix form usually makes it both clear and quick.

What is the difference between a vector, a matrix and a tensor?

A vector is a one-dimensional list of numbers, a matrix is a two-dimensional grid, and a tensor is the general name for grids of any number of dimensions. A single table row is a vector, a whole table is a matrix, and a batch of tables, or the attention scores for many heads, is a tensor.

What is matrix multiplication actually doing?

It combines two transformations, and at the level of numbers it computes a dot product between each row of the first matrix and each column of the second. In a network layer, it takes an input vector and produces a new vector in which each entry is a weighted sum of the inputs. That is how a layer mixes information.

What are eigenvalues and eigenvectors good for?

Eigenvectors are directions that a matrix only stretches, and eigenvalues are the stretch factors. For a covariance matrix, they reveal the main directions of variation in a table and how much each one matters. This underlies principal component analysis, spectral clustering and the analysis of how stable a training process is.

What is the singular value decomposition used for?

It breaks any matrix into rotations and stretches, and keeping only the largest stretches gives the best low-rank approximation. People use it to compress tables, remove noise, fill in missing entries, find main directions of variation and solve least squares problems stably. It is one of the most broadly useful tools in numerical computing.

What is rank and why does it matter?

Rank counts the independent directions in a matrix. A low-rank table has redundancy that can be exploited: it can be stored with fewer numbers, imputed from partial data and modeled with fewer parameters. Low-rank updates are also the basis of efficient fine-tuning methods that change a large model with only a small number of new weights.

How do tabular foundation models use linear algebra?

They convert every cell into a vector with linear maps and lookup tables, then use attention, which is built from matrix products, to mix information across columns within a row and across rows in the context. The outputs are produced by further linear layers. Efficiency tricks such as low precision and fused kernels are also linear algebra engineering.

Why does attention become expensive for large tables?

Attention compares every item with every other item, which produces a matrix whose size grows with the square of the number of items. For a table with many rows, that matrix can exhaust memory and time. Solutions include attending to subsets, processing in chunks, using approximations and using kernels that avoid storing the full matrix.

Do I need to know linear algebra to use these models?

You can use them without it, but a working grasp helps you reason about shapes, scaling, memory and why some choices matter. Understanding vectors, dot products, projections and low-rank structure makes it much easier to debug pipelines, interpret embeddings and judge whether a proposed change is sensible.

Glossary

• Attention: a way of mixing vectors using similarity scores computed from matrix products.
• Basis: a set of independent directions that can express every vector in a space.
• Condition number: a measure of how sensitive a problem is to small changes in the data.
• Covariance matrix: a matrix of how pairs of columns vary together.
• Dot product: the sum of element-wise products of two vectors.
• Eigenvalue: the factor by which a matrix stretches an eigenvector.
• Eigenvector: a direction that a matrix only stretches or flips.
• Embedding: a vector that represents a cell, a row, a column or a word.
• Identity matrix: a square matrix of ones on the diagonal and zeros elsewhere.
• Low-rank approximation: a matrix built from only the strongest directions of another.
• Norm: a measure of vector length.
• Orthogonal: at right angles, with zero dot product.
• PCA: a method that finds the directions of greatest variance.
• Projection: the part of a vector that lies along chosen directions.
• Rank: the number of independent directions in a matrix.
• SVD: a factorization of any matrix into rotations and stretches.
• Tensor: a grid of numbers with any number of dimensions.
• Transpose: a matrix with rows and columns swapped.

Summary

Linear algebra treats data as vectors and matrices and models as transformations of them. Dot products measure similarity, matrix products compose transformations, least squares fits linear models, and eigenvectors and the SVD reveal and compress structure. Careful handling of shapes, scaling, rank and numerical stability turns these ideas into reliable code.

For tabular foundation models, linear algebra is the substance of the architecture. Cells become vectors, mixed types share one space, attention across rows and columns is built from matrix products, and imputation, anomaly detection and semantic matching are geometry in that space. Efficiency work, from low precision to fused kernels to low-rank updates, is linear algebra engineering. Related pages: Optimization, Statistics, Transformers, TabPFN and /model/benchmarks.

Related

It belongs to Foundations in my stack, under AI and machine learning. Others in the same group: