Embeddings
Embeddings are numeric vectors that capture the meaning of text, images or records, so similar items sit close together. Learn how they power search, RAG and clustering.
Overview
An embedding is a list of numbers that represents a piece of data, such as a sentence, an image, a product or a user, in a way that captures its meaning. An embedding model reads the input and outputs a fixed-length vector, often a few hundred to a few thousand numbers long. The model is trained so that items with similar meaning get vectors that point in similar directions. "How do I reset my password?" and "I forgot my login details" end up close together, even though they share almost no words.
Once data is turned into vectors, many hard problems become geometry. Search becomes "find the nearest vectors to the query vector". Grouping becomes clustering points in space. Spotting duplicates becomes finding pairs that are very close. This is why embeddings sit underneath semantic search, retrieval-augmented generation, recommendation systems and many classification pipelines.
A useful analogy is a map of a city. Every shop has coordinates. Shops that sell similar things tend to cluster in the same streets: bakeries near cafés, hardware stores near builders' merchants. An embedding is a set of coordinates on a map of meaning, except the map has hundreds of dimensions instead of two.
What it is
An embedding system has a few main parts.
| Part | Role |
|---|---|
| Embedding model | A neural network that maps input to a vector |
| Vector | The fixed-length list of numbers produced for one item |
| Dimension | The length of the vector, for example 384, 768 or 1,536 |
| Similarity measure | How closeness is computed, usually cosine similarity or dot product |
| Vector index | A data structure for fast nearest-neighbour search, such as HNSW |
| Vector store | A database or library that holds vectors with metadata, such as Qdrant or pgvector |
Embeddings come in several kinds. Dense embeddings use every dimension and capture meaning broadly. Sparse embeddings have mostly zero values and behave more like weighted keywords. Multi-vector embeddings keep one vector per token, as in ColBERT. Multimodal embeddings place images and text in the same space, as in CLIP.
Where it came from
The idea that words can be represented by vectors is old, going back to distributional semantics: a word is known by the company it keeps. The modern era began with word2vec, published by a Google team led by Tomas Mikolov in 2013, which learned word vectors by predicting neighbouring words. GloVe from Stanford followed in 2014. These are covered on the word embeddings page.
Word vectors gave one vector per word regardless of context. Transformer models such as BERT in 2018 produced contextual vectors, where "bank" in "river bank" differs from "bank" in "bank account". In 2019 Sentence-BERT, by Nils Reimers and Iryna Gurevych, fine-tuned BERT-style models so that whole sentences could be compared efficiently with cosine similarity. This made sentence embeddings practical for search.
Later milestones included contrastive training on large sets of text pairs, joint image and text embeddings with CLIP in 2021, and the Massive Text Embedding Benchmark (MTEB), introduced in 2022, which gave a shared way to compare models across many tasks. Matryoshka representation learning, also published in 2022, trained vectors whose first dimensions are useful on their own, so they can be shortened to save space. Today embedding models are offered both as open-weight models and as hosted APIs.
How it works
Step 1: Prepare the input
Text is split into tokens by a tokenizer. Long documents are usually divided into chunks first, because embedding models have an input limit and because one vector for a whole book would blur its meaning.
Step 2: Encode with a neural network
The tokens pass through a transformer encoder. Each token gets a contextual vector. These are combined into one vector for the whole input, often by averaging, called mean pooling, or by taking a special token's vector.
Step 3: Normalise
Many systems scale each vector to length one. After normalisation, cosine similarity and dot product give the same ranking, and comparisons become simple.
Step 4: Train for similarity
Embedding models are usually trained with contrastive learning. The model sees pairs that should be close, such as a question and its answer, and many items that should be far apart. A loss such as InfoNCE pulls matching pairs together and pushes others away.
Step 5: Index and search
Vectors are stored in an index. At query time the query is embedded with the same model, and the index returns the nearest stored vectors. Exact search compares against everything. Approximate methods such as HNSW or IVF-PQ trade a little accuracy for large speed gains.
A short example with the open-source sentence-transformers library:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
docs = ["Reset your password from the login page.", "Our shop opens at 9am."]
doc_vecs = model.encode(docs)
query_vec = model.encode(["I forgot my login details"])
print(model.similarity(query_vec, doc_vecs)) # higher score = closer meaning
Key concepts and terms
| Term | Meaning |
|---|---|
| Vector space | The many-dimensional space in which embeddings live |
| Cosine similarity | Similarity based on the angle between two vectors |
| Dot product | Sum of element-wise products; equals cosine for unit vectors |
| Dense vector | A vector where most values are non-zero |
| Sparse vector | A vector where most values are zero, like weighted keywords |
| Chunking | Splitting long text into pieces before embedding |
| Pooling | Combining token vectors into one vector |
| Approximate nearest neighbour | Fast search that may miss a few true neighbours |
| Recall at k | Share of true matches found in the top k results |
| Matryoshka embedding | A vector that can be truncated and still work |
A worked example
An illustrative company has a help centre with 5,000 articles and wants better search.
- Chunking. Each article is split into passages of about 300 words. This gives roughly 40,000 chunks.
- Embedding. Each chunk is encoded by a model that outputs 768-dimensional vectors. Stored as 32-bit floats, one vector takes 768 times 4 bytes, about 3 KB. All chunks take about 123 MB.
- Indexing. The vectors go into an HNSW index with each chunk's article ID and language as metadata.
- Querying. A user types "card got declined abroad". The query is embedded and the index returns the 20 nearest chunks in a few milliseconds.
- Reranking. A cross-encoder rescores those 20 and keeps the best 5.
- Measuring. On 300 labelled test queries, the team measures recall at 10. Keyword search alone found the right article in the top 10 for about 62 per cent of queries; the hybrid approach reached about 85 per cent.
These figures are for illustration only. The pattern of chunk, embed, index, retrieve, rerank and measure is the standard one.
Strengths and limitations
Strengths:
- Find matches by meaning, not just shared words.
- One model can serve search, clustering, classification and deduplication.
- Work across languages with multilingual models.
- Fast at scale with approximate indexes.
- Can link different data types, such as images and captions.
Limitations:
- Weak on exact matches such as product codes, names and numbers.
- Quality depends on how well the training data matches your domain.
- Changing the model means re-embedding everything.
- Vectors are hard to interpret.
- Storage grows with dimension and corpus size.
- Similar in meaning is not the same as relevant or correct.
How it compares with alternatives
| Approach | How it matches | Strength | Weakness |
|---|---|---|---|
| Dense embeddings | Meaning in vector space | Paraphrases, synonyms | Exact terms, rare words |
| BM25 keyword search | Weighted term overlap | Exact words, codes, names | Synonyms and paraphrase |
| Hybrid search | Dense plus keyword, fused | Best of both | More moving parts |
| Multi-vector, for example ColBERT | Token-level vector matching | High precision | More storage and compute |
| Cross-encoder reranking | Reads query and document together | Highest accuracy per pair | Too slow for full corpus |
Practical guidance and best practices
- Evaluate candidate models on your own data, not only on public leaderboards.
- Use the same model, version and preprocessing for documents and queries.
- Respect query and document prefixes or instructions if the model requires them.
- Choose chunk sizes deliberately and keep titles or context with each chunk.
- Normalise vectors and use the similarity measure the model was trained for.
- Combine with keyword search for names, codes and exact phrases.
- Store the model name and version beside each vector so re-indexing is traceable.
- Consider shorter or quantised vectors to cut storage, and measure the recall cost.
Common mistakes
- Mixing vectors from different models in the same index.
- Embedding whole long documents as one vector and losing detail.
- Judging quality by a few hand-picked queries instead of a labelled set.
- Assuming the nearest result is always relevant, with no score threshold.
- Forgetting to re-embed after changing the model or chunking rules.
- Ignoring metadata filters and returning results from the wrong tenant or language.
Frequently asked questions
What dimension should I choose?
Use what the model provides, unless it supports truncation. Larger dimensions can hold more detail but cost more storage and compute. Many models work well at 384 to 1,024 dimensions. Test smaller sizes if storage matters.
Cosine similarity or dot product?
If vectors are normalised to unit length, both give the same ranking. Otherwise use the measure the model was trained with, which its documentation will state.
Can I use one embedding model for every language?
Multilingual models map many languages into one space, so a question in Turkish can match a document in English. Quality varies by language, so test on the languages you care about.
Do I need a vector database?
Not always. For a few thousand vectors, a library such as FAISS or a plain matrix search is enough. A database helps when you need persistence, filtering, updates and scale.
How do embeddings relate to large language models?
Both are often transformers, but they are trained for different goals. Embedding models output a vector for comparison. Generative models output text. In retrieval-augmented generation, embeddings find the passages and a generative model writes the answer.
Are embeddings private?
Embeddings are not plain text, but research has shown that text can sometimes be partly reconstructed from them. Treat them as sensitive if the source data is sensitive.
Glossary
- Encoder: a network that turns input into a representation.
- Vector: an ordered list of numbers.
- Nearest neighbour: the stored item closest to a query.
- Index: a structure that speeds up search.
- Normalisation: scaling a vector to length one.
- Contrastive loss: a training objective that pulls matches together and pushes non-matches apart.
- Metadata: extra fields stored with a vector, such as date or source.
- Re-embedding: recomputing all vectors after a model change.
Summary
Embeddings turn text, images and records into vectors so that similar meaning becomes nearby points in space. They grew from word2vec in 2013 through contextual transformer encoders and Sentence-BERT in 2019 to today's contrastively trained, often multilingual and multimodal models. They power semantic search, retrieval-augmented generation, clustering and deduplication. They work best with careful chunking, consistent models, hybrid keyword search for exact terms, reranking for precision and evaluation on your own data.
Related
It belongs to LLM concepts in my stack, under Generative AI and LLMs. Others in the same group: