# Word embeddings (Word2Vec, GloVe)

Word embeddings represent words as vectors so similar words are close together. Learn Word2Vec, GloVe, analogies and how they led to contextual embeddings.

## Overview

A word embedding is a list of numbers, a vector, that represents a word. The vectors are learned from large amounts of text so that words used in similar ways end up with similar vectors. "Coffee" sits near "tea", "Paris" near "Lyon", and "run" near "jog". Distances and directions in this space carry meaning that a computer can calculate with.

Before embeddings, programs usually treated each word as a separate symbol, for example a one-hot vector with a single 1 in a list as long as the vocabulary. Such vectors say nothing about which words are related: "cat" is as far from "kitten" as it is from "carburettor". Embeddings replaced these sparse, meaningless codes with short, dense vectors, typically 100 to 300 numbers, that capture similarity. Methods such as Word2Vec and GloVe made this practical and popular in the early 2010s, and they paved the way for today's [embeddings](/tools/embeddings) of sentences, documents, images and tables.

An everyday comparison is a map of a city's shops. Shops that sell similar things tend to cluster together: bakeries near cafés, hardware stores near garden centres. You could guess what a new shop sells by looking at its neighbours. Word embeddings build such a map for words, learned purely from which words appear near each other in text.

## What it is

Word embeddings rest on a small set of ideas.

| Concept | Meaning |
|---|---|
| Distributional hypothesis | Words that appear in similar contexts tend to have similar meanings |
| Vocabulary | The set of words that receive vectors |
| Embedding matrix | A table with one row per word and one column per dimension |
| Context window | The neighbouring words used to define a word's context |
| Skip-gram | Word2Vec variant that predicts context words from a centre word |
| CBOW | Continuous bag of words: predicts the centre word from its context |
| Negative sampling | Training trick that contrasts real word pairs with random ones |
| GloVe | Method that fits vectors to global word co-occurrence statistics |
| fastText | Builds word vectors from character n-grams, so it handles unseen words |
| Cosine similarity | The usual measure of how close two vectors are in direction |

These are static embeddings: each word has exactly one vector, whatever sentence it appears in. Later contextual models such as [BERT](/tools/bert) produce a different vector for each occurrence, so "bank" in "river bank" and in "bank account" get different representations.

## Where it came from

The linguistic idea is old. In the 1950s, Zellig Harris argued that meaning relates to distribution, and J. R. Firth summed it up in 1957 as "you shall know a word by the company it keeps". Count-based methods such as latent semantic analysis in the late 1980s and 1990s built word vectors from matrices of word and document counts.

Neural approaches followed. In 2003, Yoshua Bengio and colleagues trained a neural language model that learned word vectors as part of predicting the next word. The big shift came in 2013, when Tomas Mikolov and colleagues at Google released Word2Vec. Its skip-gram and CBOW models, trained with tricks such as negative sampling, were fast enough to learn from billions of words on ordinary hardware, and the vectors showed striking regularities, such as the vector for "king" minus "man" plus "woman" landing near "queen".

In 2014, Jeffrey Pennington, Richard Socher and Christopher Manning at Stanford introduced GloVe, which learns vectors from global co-occurrence counts. In 2016 and 2017, researchers at Facebook AI Research, including Piotr Bojanowski and Tomas Mikolov, released fastText, which represents words as bags of character n-grams. From 2018, contextual models such as ELMo and BERT, and later [transformer](/tools/transformers)-based embedding models, took over most applications, but static embeddings remain a clear and cheap way to understand the core idea.

## How it works

### Word2Vec skip-gram, step by step

1. **Slide a window.** Move through the text one word at a time. For each centre word, take the words within a window, say two words either side, as its context.
2. **Make pairs.** From "the quick brown fox jumps", with "brown" as centre, you get pairs such as (brown, quick), (brown, fox), (brown, the), (brown, jumps).
3. **Score pairs.** Each word has two vectors: one for when it is the centre and one for when it is context. The score of a pair is the dot product of the two vectors.
4. **Contrast with noise.** For each real pair, also draw a few random "negative" words, perhaps five. Train so real pairs get high scores and random pairs get low scores, using a logistic loss.
5. **Update.** [Backpropagation](/tools/backpropagation) adjusts the vectors. Over billions of pairs, words that share contexts are pulled towards similar vectors.
6. **Keep the result.** After training, the centre-word vectors are usually kept as the embeddings.

CBOW reverses the task, averaging context vectors to predict the centre word. It is faster; skip-gram tends to do better on rare words.

### GloVe

GloVe first counts, across the whole corpus, how often each word appears near each other word. It then learns vectors whose dot products match the logarithm of those counts, with a weighting that stops very common pairs from dominating. The idea is that ratios of co-occurrence probabilities carry meaning, and the vectors are fitted to reproduce them.

### fastText

fastText breaks each word into character n-grams, such as "whe", "her" and "ere" for "where", plus the word itself. A word's vector is the sum of its n-gram vectors. This helps with rare words, typos and languages with rich word forms, and it can build a vector for a word never seen in training.

### Using the vectors

Similarity is measured with cosine similarity. Analogies are explored with vector arithmetic. For downstream tasks, embeddings can be fed into classifiers, or used to initialise the first layer of a larger network.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Embedding | A dense vector representing an item |
| Dimension | One of the numbers in the vector; typical sizes are 100 to 300 |
| One-hot vector | A sparse vector with a single 1, used before embeddings |
| Co-occurrence | Two words appearing near each other in text |
| Skip-gram | Predicting context words from a centre word |
| CBOW | Predicting a centre word from surrounding words |
| Negative sampling | Using random words as counter-examples during training |
| Cosine similarity | Similarity based on the angle between vectors |
| Out-of-vocabulary word | A word without a learned vector |
| Static embedding | One vector per word regardless of context |
| Contextual embedding | A vector that depends on the surrounding sentence |

## A worked example

This illustrative example uses made-up three-dimensional vectors so the arithmetic is visible. Real embeddings have hundreds of dimensions.

| Word | Vector |
|---|---|
| king | (0.8, 0.7, 0.1) |
| queen | (0.8, 0.1, 0.7) |
| man | (0.2, 0.8, 0.1) |
| woman | (0.2, 0.2, 0.7) |

1. Compute king − man + woman: (0.8 − 0.2 + 0.2, 0.7 − 0.8 + 0.2, 0.1 − 0.1 + 0.7) = (0.8, 0.1, 0.7).
2. The result equals the vector for queen in this toy table. In real embeddings it lands near, not exactly on, the queen vector.
3. Cosine similarity of man and woman: dot product = 0.04 + 0.16 + 0.07 = 0.27. Lengths ≈ 0.83 and 0.75. Similarity ≈ 0.27 / 0.62 ≈ 0.43.

A practical scenario, also illustrative: a support team wants to route 50,000 customer emails into ten topics. They average pretrained word vectors for each email, train a [logistic regression](/tools/logistic-regression) classifier on 3,000 labelled examples and get a fast, explainable baseline. Later they compare it with a sentence embedding model and keep whichever does better on held-out emails.

## Strengths and limitations

**Strengths:**

- Captures word similarity in compact vectors.
- Cheap to train and to use, even on a laptop.
- Pretrained vectors exist for many languages.
- Easy to inspect: nearest neighbours and analogies are intuitive.
- fastText handles rare and unseen words.

**Limitations:**

- One vector per word cannot separate different senses of the same word.
- Word order and sentence structure are mostly lost when vectors are averaged.
- Vectors absorb social biases present in the training text.
- Analogy results are less reliable than early demonstrations suggested.
- Largely superseded by contextual models for demanding tasks.

## How it compares with alternatives

| Method | Representation | Context-aware | Notes |
|---|---|---|---|
| One-hot or bag of words | Sparse counts | No | No notion of similarity |
| Word2Vec | Dense, predictive | No | Fast, popularised embeddings |
| GloVe | Dense, count-based | No | Fits global co-occurrence |
| fastText | Dense, subword-based | No | Handles unseen words |
| [BERT](/tools/bert) and transformer [embeddings](/tools/embeddings) | Dense, contextual | Yes | State of the art for most tasks |

## Practical guidance and best practices

1. Use pretrained vectors unless you have a large, specialised corpus.
2. Train your own vectors when your domain uses words in unusual ways, such as medical or legal text.
3. Use the same tokenisation and lower-casing as the vectors were trained with.
4. Prefer fastText when your text has many rare words, misspellings or rich word forms.
5. Normalise vectors to unit length before computing cosine similarity at scale.
6. Evaluate on your actual task, not only on analogy tests.
7. Check vectors for unwanted associations before using them in decisions about people.
8. Consider modern sentence embedding models when meaning depends on full sentences.

## Common mistakes

- Treating analogy arithmetic as reliable proof that the vectors "understand" meaning.
- Ignoring words missing from the vocabulary, which silently drops information.
- Averaging vectors for long documents and expecting order-sensitive meaning to survive.
- Using embeddings trained on general web text for a narrow technical domain without checking.
- Assuming embeddings are neutral when they reflect patterns, including biases, in their training data.

## Frequently asked questions

### What is the difference between Word2Vec and GloVe?

Word2Vec learns by predicting nearby words in a sliding window. GloVe first counts co-occurrences across the whole corpus and then fits vectors to those counts. In practice they produce embeddings of similar quality, and the better choice depends on the data and task.

### Are word embeddings still used?

Yes, in lightweight systems, search features, teaching and as baselines. For most demanding language tasks, contextual embeddings from transformer models have replaced them.

### Why does king − man + woman give queen?

Training places words so that consistent relationships, such as male to female, appear as roughly consistent directions. Adding and subtracting those directions moves you to a related word. The effect is real but approximate, and it does not hold for every analogy.

### How many dimensions should an embedding have?

Common choices are 100 to 300 for static word embeddings. More dimensions can capture finer distinctions but need more data to train well. Testing a few sizes on your task is the reliable way to choose.

### Do word embeddings contain bias?

Yes. Because they learn from human-written text, they reflect associations present in it, including stereotypes. Researchers have documented gender and other biases in widely used vectors, so they should be audited before use in sensitive applications.

## Glossary

- Analogy: a relationship such as "a is to b as c is to d", tested with vector arithmetic.
- Corpus: the body of text used for training.
- Dense vector: a vector where most values are non-zero.
- Distributional semantics: the study of meaning through patterns of word use.
- N-gram: a sequence of n characters or words.
- Nearest neighbour: the vector closest to a given vector.
- Token: a unit of text, such as a word or word piece.

## Summary

Word embeddings represent words as dense vectors learned from text, placing words that share contexts close together. Word2Vec in 2013, GloVe in 2014 and fastText soon after turned an old linguistic idea into fast, practical tools and showed that meaning could be captured in geometry. Static embeddings have known limits, including a single sense per word and inherited bias, and contextual transformer models now lead, but the core idea underlies every modern embedding.