Vespa

Overview

Vespa is an open-source platform for applications that need to search, rank and serve large amounts of data with low latency. It combines full-text search, structured filtering, vector search and machine-learned ranking in one engine. Instead of fetching candidates from a database and ranking them in a separate service, Vespa evaluates ranking models next to the data, on the nodes that store it. It is released under the Apache 2.0 licence, written mainly in Java and C++, and developed by Vespa.ai, which was spun out of Yahoo in 2023.

Vespa's distinctive idea is that ranking is a first-class part of the engine. You describe how each result should be scored in a ranking profile, using text signals such as BM25, vector similarity, document attributes and models such as gradient-boosted trees or neural networks. Vespa computes these scores in stages, cheaply for many candidates and more expensively for the best few. This makes it suited to search engines, recommendation systems, advertising and large retrieval-augmented generation (RAG) systems.

Think of a large restaurant kitchen during service. Rather than sending every order to one head chef, the kitchen has stations that prepare, check and plate dishes right where the ingredients are stored, and the head chef only does the final touches. Vespa works like this: scoring happens where the data lives, and only the best candidates move on for expensive final ranking.

What it is

Vespa can be self-hosted, from a single container to large clusters, or used through the managed Vespa Cloud service.

Where it came from

Vespa's roots go back to a web search engine started in 1997 in Trondheim, Norway. The technology came to Yahoo through its acquisition of Overture in 2003, and was developed inside Yahoo for many years to power search, recommendation, personalised content and advertising. Yahoo released Vespa as open source in 2017.

On 4 October 2023 Yahoo spun Vespa out as an independent company, Vespa.ai, with Jon Bratseth as chief executive and Yahoo remaining a customer and investor. In November 2023 the new company announced a 31 million US dollar Series A investment led by Blossom Capital. Over the same years Vespa added features aimed at modern AI retrieval: approximate nearest-neighbour search with HNSW, built-in embedding models, multi-vector and late-interaction ranking similar to ColBERT, and global-phase ranking across all content nodes. It is used both inside Yahoo and by other companies for search and recommendation at large scale.

How it works

Step 1: Define the application

You write a schema for each document type. For a product catalogue you might declare a title as indexed text, a price as an attribute, and an embedding as a tensor field with an HNSW index. You also write ranking profiles. All of this, plus any model files, goes into an application package.

Step 2: Deploy

The package is deployed to the config server, which distributes configuration to container and content nodes. Changing a schema or ranking profile is a redeployment, and many changes take effect without downtime.

Step 3: Feed documents

Documents are sent to container nodes, which can run processing such as embedding text into vectors. They are then distributed across content nodes with a chosen redundancy, so each document is stored on more than one node.

Step 4: Match

A query is written in YQL and can combine text matching, filters and nearest-neighbour search. Content nodes find matching documents in parallel. A hybrid query might look like this:

select * from product where ({targetHits: 200}nearestNeighbor(embedding, q)) or userQuery()

Step 5: Rank in phases

Each content node scores its matches with the ranking profile's first phase, a cheap expression such as a mix of BM25 and vector closeness. The top results on each node are rescored in a second phase with a costlier model, for example a gradient-boosted tree from XGBoost or LightGBM, or an ONNX neural model. An optional global phase reranks the merged top results across all nodes, which suits heavy models such as cross-encoders.

Step 6: Group and return

Container nodes merge results, apply grouping and aggregation if requested, and return the final list with summaries. Custom Java components can add logic, such as calling a language model for RAG.

Key concepts and terms

A worked example

An illustrative news site wants article search and "more like this" recommendations over 30 million articles, with fresh content arriving every minute.

1. The schema has title and body as indexed text, publish time and section as attributes, and a 384-dimensional embedding tensor with an HNSW index. The embedder runs on container nodes at feed time.
2. The first-phase ranking combines BM25 on title and body with the closeness of the embedding to the query embedding, plus a freshness term based on publish time.
3. The second phase applies a LightGBM model trained on click data, using about 40 features including those signals and article popularity, to the top 200 matches on each content node.
4. A global phase applies a small cross-encoder to the top 50 overall.
5. A query such as "flood warnings in the north" matches by text and by meaning, filters to the last 7 days, and returns 10 results in tens of milliseconds in this setup.
6. New articles are searchable within seconds of being fed, without rebuilding indexes.
7. The team tests a new ranking model by deploying a second ranking profile and sending a share of traffic to it.

All figures are illustrative. The pattern, staged ranking next to the data with real-time updates, is what Vespa is designed for.

Strengths and limitations

Strengths:

• Text search, filtering, vector search and machine-learned ranking in one engine.
• Multi-phase ranking keeps heavy models affordable.
• Real-time writes and updates, including partial updates of fields.
• Proven at very large scale over many years at Yahoo.
• Supports tensors, ONNX models, tree models and multi-vector ranking.
• Open source under Apache 2.0, with a managed cloud option.

Limitations:

• Steeper learning curve than simple vector databases.
• Concepts such as application packages, schemas and ranking expressions take time to learn.
• Self-hosting large clusters needs operational expertise.
• Overkill for small prototypes that need only basic vector search.
• Smaller general community than some mainstream databases, though documentation is extensive.

How it compares with alternatives

Practical guidance and best practices

1. Start with the sample applications and a single-node setup before designing a cluster.
2. Design schemas around queries: decide which fields are indexed text, which are attributes and which are tensors.
3. Keep first-phase ranking cheap and push expensive models to later phases.
4. Use learning to rank with real interaction data to train second-phase models.
5. Combine BM25 and vector closeness for hybrid search, and tune weights on labelled queries.
6. Use separate ranking profiles to test new models side by side.
7. Monitor latency per phase and adjust how many candidates move between phases.
8. Use partial updates for fast-changing fields such as stock or popularity.

Common mistakes

• Treating Vespa as a plain vector store and ignoring its ranking features.
• Putting a heavy neural model in the first phase and overloading content nodes.
• Making every field an attribute and using more memory than needed.
• Skipping a labelled evaluation set, so ranking changes cannot be judged.
• Setting targetHits too low for nearest-neighbour search and losing recall.
• Underestimating the learning curve and planning too little time for schema design.

Frequently asked questions

Is Vespa free?

Yes. Vespa is open source under the Apache 2.0 licence and can be self-hosted at no licence cost. Vespa Cloud, the managed service, is paid.

Is Vespa a vector database?

It includes vector search, but it is broader. It is a serving engine that combines text search, filtering, vector search and machine-learned ranking. Many teams use it where a vector database alone would need extra ranking services.

What are tensors in Vespa?

Tensors are typed multi-dimensional values stored in documents or sent with queries. Embeddings are one-dimensional tensors, and ranking expressions can combine tensors with other features and models.

Can Vespa run machine learning models?

Yes. Ranking profiles can evaluate ONNX models, gradient-boosted tree models such as XGBoost and LightGBM, and custom expressions, close to the data.

Is Vespa good for RAG?

It is a strong option for large RAG systems that need hybrid retrieval and careful ranking. For small prototypes, simpler tools may be faster to set up.

Who maintains Vespa?

Vespa.ai, the company spun out of Yahoo in 2023, leads development, with contributions from the open-source community.

Glossary

• Ranking: ordering results by a computed score.
• Ranking expression: a formula that combines features into a score.
• Feature: a signal used in ranking, such as BM25 or closeness.
• Redundancy: the number of copies of each document in a cluster.
• Partial update: changing some fields of a document without rewriting it.
• Grouping: aggregating results, for example counts per category.
• Container: a stateless node running query and document processing.
• Embedder: a component that turns text into vectors inside Vespa.

Summary

Vespa is an open-source engine for search, recommendation and vector retrieval, with roots in a 1997 search engine, years of development at Yahoo, open-sourcing in 2017 and spin-out as Vespa.ai in October 2023. It combines text matching, filtering, nearest-neighbour search and machine-learned ranking, and evaluates ranking in phases on the nodes that hold the data. This makes it powerful for large, latency-sensitive applications where ranking quality matters, at the cost of a steeper learning curve than simpler vector databases.

Related

It belongs to Vector databases in my stack, under Generative AI and LLMs. Others in the same group: