Principal component analysis (PCA)
Principal component analysis finds the directions of greatest variance in data and projects onto a few of them. Learn how PCA works and when to use it.
Overview
Principal component analysis, or PCA, takes data with many correlated features and replaces them with a smaller set of new features, called principal components, that keep as much of the original variation as possible. The first component is the single direction along which the data spread out the most. The second is the direction of greatest remaining spread at right angles to the first, and so on. Keeping only the first few components often preserves most of the information in a fraction of the space.
PCA is used to compress data, to remove noise, to speed up other models, and to draw high-dimensional data on a two-dimensional chart. It is an unsupervised learning method: it looks only at the features, never at a target label.
An everyday picture is photographing a three-dimensional object. A flat photo loses one dimension, but if you choose the angle that shows the object's longest and widest sides, the photo still tells you most of what matters. PCA chooses that best angle mathematically, in any number of dimensions.
What it is
| Part | What it means |
|---|---|
| Principal component | A new axis, a weighted combination of the original features |
| Loadings | The weights that define each component in terms of the original features |
| Scores | The coordinates of each example on the new axes |
| Explained variance | How much of the total spread each component captures |
| Explained variance ratio | That amount as a share of the total |
| Covariance matrix | A table of how each pair of features varies together |
| Eigenvector and eigenvalue | The direction of a component and the variance along it |
Principal components are uncorrelated with one another and are ordered by importance. If the first three components explain 90 per cent of the variance, you can describe each example with three numbers instead of, say, fifty, and lose only 10 per cent of the spread.
PCA is linear. It can rotate and project, but it cannot unfold curved structures. For visualising non-linear clusters, methods such as t-SNE and UMAP are often better, though they serve a different purpose.
Where it came from
Karl Pearson introduced the idea in 1901 in a paper on fitting lines and planes that come closest to a set of points in space. Harold Hotelling developed the method independently in 1933, in the context of analysing many related variables in educational psychology, and gave it the name "principal components". The same mathematics appears in signal processing under the name Karhunen-Loève transform, and it is closely tied to the singular value decomposition from linear algebra.
PCA became practical on large datasets with computers, and it spread into almost every quantitative field: psychology, chemistry, genetics, finance, image processing and climate science. Extensions followed, including kernel PCA for non-linear structure, sparse PCA for more interpretable loadings, incremental PCA for data that do not fit in memory, and randomised algorithms that approximate the top components quickly. It remains a standard tool in libraries such as scikit-learn.
How it works
Step 1: centre and usually scale
Subtract each feature's mean so the data are centred at zero. If features are in different units, such as euros and years, also divide by the standard deviation. Otherwise the feature with the largest numbers dominates the components.
Step 2: measure how features vary together
Conceptually, compute the covariance matrix, which records how each pair of features moves together. Strong correlations mean there is redundancy that PCA can compress.
Step 3: find the main directions
Find the eigenvectors and eigenvalues of the covariance matrix. Each eigenvector is a component direction; its eigenvalue is the variance along it. In practice, libraries compute the same result more stably with the singular value decomposition of the centred data.
Step 4: choose how many components to keep
Sort components by variance and look at the cumulative explained variance. Common rules are to keep enough components for 90 or 95 per cent of the variance, or to look for an "elbow" in a scree plot where extra components add little. If PCA feeds a model, choose the number by cross-validation.
Step 5: project the data
Multiply the centred data by the chosen eigenvectors to get the scores. These are the new, lower-dimensional features.
Step 6: reconstruct if needed
You can map scores back to the original space. The reconstruction error shows what was lost, and unusually large reconstruction error is a simple signal for anomaly detection.
from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
pca = make_pipeline(StandardScaler(), PCA(n_components=0.95))
X_reduced = pca.fit_transform(X_train) # keeps 95% of the variance
Key concepts and terms
| Term | Meaning |
|---|---|
| Dimensionality reduction | Representing data with fewer features |
| Variance | How spread out values are |
| Covariance | How two features vary together |
| Eigenvector | A direction left unchanged in orientation by a matrix transformation |
| Eigenvalue | The variance captured along an eigenvector |
| Singular value decomposition | A factorisation used to compute PCA stably |
| Scree plot | A chart of variance per component, used to choose how many to keep |
| Loadings | How much each original feature contributes to a component |
| Whitening | Rescaling components to unit variance |
| Reconstruction error | The difference between original data and data rebuilt from components |
| Kernel PCA | A non-linear extension using kernel functions |
A worked example
This scenario is illustrative. A food lab measures 40 chemical properties of 500 olive oil samples. Many of the properties are strongly correlated, and the team wants to see whether oils from different regions form groups.
After standardising the 40 features, PCA shows that the first component explains 38 per cent of the variance, the second 21 per cent and the third 11 per cent. The first five together explain 84 per cent, and the first ten explain 95 per cent.
Plotting each sample on the first two components reveals three loose groups that match three growing regions. The loadings show that the first component is driven mainly by several fatty-acid measures, and the second by two phenolic compounds. The team then trains a logistic regression on the first ten components to classify region. Cross-validated accuracy is 91 per cent, compared with 92 per cent on all 40 features, with a model that is faster to train and less affected by correlated inputs.
Strengths and limitations
Strengths:
- Fast, deterministic and well understood.
- Removes redundancy between correlated features.
- Reduces noise by discarding low-variance directions.
- Produces uncorrelated features, which helps some models.
- New data can be projected with the same transformation.
Limitations:
- Captures only linear structure.
- Components can be hard to interpret, as each mixes many features.
- Sensitive to scaling choices and to outliers.
- High variance does not always mean useful for prediction; a low-variance direction can matter most for the target.
- Not designed to preserve cluster structure for visualisation.
How it compares with alternatives
| Method | Linear | Keeps global structure | Transforms new data | Main use |
|---|---|---|---|---|
| PCA | Yes | Yes | Yes | Compression, preprocessing, quick plots |
| t-SNE | No | Weakly | Not natively | Visualising local clusters |
| UMAP | No | Better than t-SNE | Yes | Visualisation and embeddings |
| Autoencoders | No | Depends on training | Yes | Non-linear compression |
| Feature selection | Not applicable | Keeps original features | Yes | Interpretable reduction |
Practical guidance and best practices
- Standardise features when they are in different units.
- Fit PCA on training data only, then apply it to validation and test data.
- Check explained variance before deciding how many components to keep.
- When PCA feeds a model, choose the number of components by cross-validation.
- Inspect loadings to understand what each top component represents.
- Handle outliers first, or consider robust PCA variants.
- Use incremental or randomised PCA for very large datasets.
- Run PCA before t-SNE or UMAP on very high-dimensional data to reduce noise and cost.
Common mistakes
- Running PCA on unscaled features with very different units.
- Fitting PCA on the full dataset before splitting, which leaks test information.
- Assuming the top components are the ones most related to the target.
- Treating clusters in a two-component plot as proof of real groups.
- Applying PCA to categorical one-hot data without thinking about what variance means there.
Frequently asked questions
How many components should I keep?
There is no universal answer. For compression, keep enough for 90 to 95 per cent of the variance. For visualisation, two or three. For a downstream model, test several values with cross-validation and pick the one that gives the best validation score.
Is PCA a feature selection method?
No. Feature selection keeps some original features and drops others. PCA creates new features that are combinations of all the original ones. If you need to keep original, named features for interpretability, use feature selection instead.
Should I always standardise before PCA?
Standardise when features are in different units or have very different scales. If all features share a unit and their natural scale matters, as with pixel intensities, centring without scaling can be appropriate.
What is the link between PCA and SVD?
The principal component directions are the right singular vectors of the centred data matrix, and the variances are the squared singular values divided by the number of examples minus one. Libraries compute PCA through SVD because it is numerically stable.
Can PCA handle missing values?
Standard PCA cannot. You need to impute missing values first, or use specialised probabilistic or iterative PCA methods that estimate components and missing entries together.
Glossary
- Principal component: a new axis capturing maximum remaining variance.
- Explained variance ratio: the share of total variance captured by a component.
- Loading: the weight of an original feature in a component.
- Score: an example's coordinate on a component.
- Centring: subtracting the mean from each feature.
- Standardisation: centring and dividing by the standard deviation.
- SVD: singular value decomposition, the usual way to compute PCA.
- Projection: mapping data onto a lower-dimensional space.
Summary
Principal component analysis finds the uncorrelated directions along which data vary most and projects the data onto the top few, compressing many correlated features into a handful of components. Introduced by Pearson in 1901 and named by Hotelling in 1933, it is fast, deterministic and widely used for compression, denoising, preprocessing and quick visualisation. It is linear, sensitive to scaling and blind to the target. Standardise, fit on training data only, check explained variance and validate the number of components for the task.
Related
It belongs to ML foundations in my stack, under Machine learning. Others in the same group: