Elasticsearch

Overview

Elasticsearch is a distributed search and analytics engine. It indexes documents so that you can find them fast by words, filters, and ranges, and so that you can compute aggregations such as counts, averages, and histograms over matching documents. It is built on the Lucene search library and is queried over a REST API with JSON.

Common uses are website and application search, log and event analysis, security analytics, and retrieval for language-model applications. It is usually not the system of record. Instead, it holds a searchable copy of data that lives elsewhere. This page explains how Elasticsearch works, how it compares with neighbors, and how it relates to tabular foundation models, which learn from raw tables of rows and columns. In that picture, Elasticsearch is a retrieval and profiling layer: it finds the right rows, searches column descriptions, and summarizes large tables quickly. For the model side, see /model/research.

What it is

Elasticsearch is a server, or a cluster of servers, that stores JSON documents in indexes. An index is similar to a table or collection. Each document is broken into fields, and each field is analyzed and stored in structures that make search fast. You send documents with a simple HTTP call and search with a JSON query language.

Main traits:

• Inverted index. A map from each term to the documents that contain it.
• Full-text search. Text is split into tokens, normalized, and scored for relevance.
• Structured search. Exact filters, ranges, and sorting on numbers, dates, and keywords.
• Aggregations. Group-bys, metrics, histograms, and percentiles over results.
• Near real-time. New documents become searchable within about a second.
• Distributed. An index is split into shards and replicated across nodes.
• Schema-flexible. Fields can be created automatically, or you can define mappings.
• Vector search. Modern versions can store embeddings and find nearest neighbors.
• Part of a stack. It is often used with tools that collect, ship, and visualize data.

Everyday analogy

Think of the index at the back of a textbook. Instead of reading every page to find "photosynthesis", you look up the word and get a list of pages. Elasticsearch builds such an index for every word in every field of every document, then adds the ability to combine words with filters, such as "pages after chapter 5 that mention photosynthesis and not chlorophyll", ranked by how well they match.

Where it came from

Lucene is a Java search library created in the late 1990s. It powers a large share of search features in software. Elasticsearch was created around 2010 as a distributed server built on top of Lucene, with a friendly JSON API, so that people did not have to build clustering, replication, and a network interface themselves. A company formed around it and added visualization, ingestion, and security tools. Over the years the project has changed its licensing, and open-source forks and compatible alternatives exist. Check current terms for the version you plan to use.

Its history shows a pattern: search engines for text grew into general tools for log analysis and metrics, because the same indexing and aggregation machinery works well on machine-generated data. More recently, vector search and hybrid retrieval were added to serve language-model applications.

How it works

Step 1: Documents and indexes

You send a JSON document to an index. Elasticsearch gives it an id and stores the original source. It also extracts each field and builds search structures for it.

PUT /products/_doc/1
{
  "name": "Trail running shoe",
  "brand": "Acme",
  "price": 79.9,
  "tags": ["running", "outdoor"],
  "in_stock": true,
  "added": "2026-02-10"
}

Step 2: Analysis turns text into terms

For a text field, an analyzer runs. It splits the text into tokens, lowercases them, can remove common words, and can reduce words to stems. "Trail running shoes" might become the terms trail, run, shoe. Queries pass through the same analysis, so "Running Shoe" matches the same terms. Think of it as a careful librarian who normalizes every title before filing it.

Step 3: The inverted index

For every term, the index lists the documents where it appears, often with positions and counts. To find documents that contain two terms, the engine intersects two sorted lists, which is very fast. Numeric and date fields use other structures that support range queries, and keyword fields store exact values for filtering and grouping.

Step 4: Segments

Lucene writes data in immutable pieces called segments. New documents go to a memory buffer, and a refresh makes them searchable by writing a new small segment. Background merges combine small segments into larger ones, and deleted documents are dropped at that time. This is why search is near real-time, and why updates are really delete-and-add operations.

Step 5: Shards and replicas

An index is split into primary shards. Each shard is a full Lucene index living on some node. Each primary can have replica shards on other nodes for safety and for extra read capacity. A query goes to every relevant shard, each shard returns its best results, and a coordinating node merges them.

Index "logs" : 5 primary shards, 1 replica each
Node A: shard 0 (p), shard 3 (r), shard 4 (p)
Node B: shard 1 (p), shard 0 (r), shard 2 (p)
Node C: shard 2 (r), shard 4 (r), shard 3 (p), shard 1 (r)

Step 6: Relevance scoring

For text queries, each matching document gets a score. The classic idea is that a term counts more when it is rare across the collection and frequent in the document, with length normalization so short fields are not unfairly penalized. A standard modern ranking function in this family is called BM25. Scores let you sort by relevance, and you can boost fields, such as titles over body text.

Step 7: Queries

The query language lets you combine conditions. A bool query has must, should, must_not, and filter parts. Filters are cached and do not affect scores.

GET /products/_search
{
  "query": {
    "bool": {
      "must":   [ { "match": { "name": "running shoe" } } ],
      "filter": [ { "range": { "price": { "lte": 100 } } },
                  { "term":  { "in_stock": true } } ]
    }
  },
  "sort": [ "_score" ],
  "size": 10
}

Step 8: Aggregations

Aggregations compute summaries over the matching documents. They can be nested, so you can group by brand and, inside each group, compute the average price.

GET /products/_search
{
  "size": 0,
  "aggs": {
    "by_brand": {
      "terms": { "field": "brand.keyword", "size": 5 },
      "aggs": { "avg_price": { "avg": { "field": "price" } } }
    }
  }
}

Aggregations use column-oriented structures built at index time, called doc values, which is why they stay fast on large data.

Step 9: Mappings

A mapping describes the fields of an index and their types. Dynamic mapping guesses types from the first value it sees, which is convenient but can guess wrong. A string may become both a text field and a keyword sub-field. A good practice is to define explicit mappings for important indexes.

Step 10: Vector and hybrid search

A dense vector field stores an embedding. Approximate nearest neighbor search, often based on graph indexes like HNSW, finds items whose vectors are close to the query vector. Combining this with keyword scores gives hybrid retrieval, which is explained in hybrid search.

Key concepts and terms

A worked example

We will index a small table of orders, search it, aggregate it, and use the result as input to a tabular model.

Create a mapping

PUT /orders
{
  "mappings": {
    "properties": {
      "order_id":   { "type": "keyword" },
      "country":    { "type": "keyword" },
      "note":       { "type": "text" },
      "total":      { "type": "float" },
      "n_items":    { "type": "integer" },
      "created_at": { "type": "date" }
    }
  }
}

Explicit types help avoid surprises later, such as a number being stored as text.

Index with the Python client

from elasticsearch import Elasticsearch, helpers

es = Elasticsearch("https://example.internal:9200", api_key="placeholder")

docs = [
    {"order_id": "o-1", "country": "DE", "note": "gift wrap please",
     "total": 59.0, "n_items": 3, "created_at": "2026-03-04T10:15:00Z"},
    {"order_id": "o-2", "country": "FR", "note": "",
     "total": 12.5, "n_items": 1, "created_at": "2026-03-05T08:00:00Z"},
]
helpers.bulk(es, ({"_index": "orders", "_id": d["order_id"], "_source": d}
                  for d in docs))

Use the bulk helper for many documents. Single-document calls are much slower at scale.

Search and aggregate

resp = es.search(index="orders", query={
    "bool": {
        "must": [{"match": {"note": "gift"}}],
        "filter": [{"range": {"total": {"gte": 20}}}]
    }
})
print(resp["hits"]["total"])

agg = es.search(index="orders", size=0, aggs={
    "by_country": {"terms": {"field": "country"},
                   "aggs": {"avg_total": {"avg": {"field": "total"}},
                            "p90": {"percentiles": {"field": "total",
                                                    "percents": [90]}}}}
})
for b in agg["aggregations"]["by_country"]["buckets"]:
    print(b["key"], b["doc_count"], b["avg_total"]["value"])

Export a flat table

To give data to a tabular model, retrieve documents in a stable order and flatten them.

import pandas as pd

rows = []
for hit in helpers.scan(es, index="orders",
                        query={"query": {"match_all": {}}},
                        _source=["order_id", "country", "total", "n_items", "created_at"]):
    rows.append(hit["_source"])
df = pd.DataFrame(rows)
print(df.dtypes)
print(df.isna().mean())

The scan helper streams all matching documents in batches. The last line prints the share of missing values per column. A document that lacks a field simply has no value for it, so missing values appear as nulls after export.

What to check

1. Are the types in the data frame the ones you expect, or did dates arrive as strings?
2. Are there fields that appear in only some documents?
3. Is the export complete? Compare the row count with the index count.
4. Are the rows in a stable order, with a key you can use to join predictions back?
5. Did you remove free text that may contain personal data before export?

Strengths and limitations

Strengths

• Very fast full-text search with relevance ranking.
• Flexible filtering, sorting, and faceting in one engine.
• Fast aggregations over large collections of documents.
• Scales out with shards and replicas.
• Near real-time indexing.
• A rich ecosystem of ingestion, visualization, and client tools.
• Handles semi-structured data and changing fields well.
• Supports vector and hybrid retrieval.

Limitations

• Not a transactional database, and not ideal as the only system of record.
• Updates are delete-and-add under the hood, so frequent updates to large documents are costly.
• Joins are limited. Data is usually denormalized.
• Heavy analytical scans cost more than in a dedicated column store.
• Mapping mistakes are hard to fix later and often need a reindex.
• Too many shards or too many fields can overload a cluster.
• Memory, especially heap and file cache, needs careful sizing.
• Deep pagination over huge result sets needs special handling.

How it compares with alternatives

A common pattern is to keep data in a durable primary store, stream changes into Elasticsearch for search, and send aggregated copies to a warehouse for reporting. Kafka Connect is one way to build that stream.

Role in tabular foundation models

A tabular foundation model learns from raw tables. Each row is a record, each column is a typed feature, and the model must handle mixed types, missing values, unclear or absent column names, and relationships that change from one table to the next. See /model/architecture. Elasticsearch is not a place where such a model runs. Its role is to find, filter, profile, and summarize tabular data, and to retrieve the right rows and the right metadata for a model.

Row and column structure

Each Elasticsearch document can be viewed as a row, and each field as a column. Doc values store each field in a column-oriented way, which is why a query can aggregate one field across millions of documents without reading everything else. A tabular export is straightforward: choose the fields, stream the documents, and flatten nested objects into dotted column names.

Mixed types

Mappings define types such as keyword, text, integer, float, date, boolean, and geo point. These types make the intent clear. A keyword field is a category, a text field is free text, and a date is a timestamp. A tabular model needs this distinction, because it treats numbers, categories, and text differently. If a mapping is wrong, for example a number mapped as text, it will mislead everything downstream.

Missing values

A document that lacks a field is not indexed for that field. You can search for it with an exists query, which makes it easy to measure missingness per column. You can also set a null value replacement in a mapping, but that hides the gap, so do so with care. Before sending data to a model, compute the share of missing values per field and look for patterns by time or category.

GET /orders/_count
{ "query": { "bool": { "must_not": { "exists": { "field": "total" } } } } }

No column names

Some indexes use cryptic field names. For a model that does not rely on names, the values are enough, but people and retrieval systems need meaning. Elasticsearch is good at searching text, so you can store a description or a data dictionary entry for each column in a separate index and search it. A query like "customer postal code" then finds the cryptic column that holds it.

Pre-training data

Pre-training of tabular foundation models relies mostly on synthetic tables and on large collections of real tables where permitted. Elasticsearch can help organize and search such collections: an index of table metadata, with column names, types, and profile statistics, lets you find tables by shape or by content and choose diverse samples. It is not the place where training itself happens.

In-context learning

A model that does in-context learning reads labeled example rows and then predicts for new rows. Choosing which rows to show matters. Retrieval is exactly what Elasticsearch does. You can find the most similar rows using filters on categorical fields, ranges on numeric ones, or a vector of the row. The top matches become the context table. Hybrid scoring can combine exact filters with vector similarity, as explained in hybrid search.

GET /labeled_rows/_search
{
  "size": 64,
  "query": {
    "bool": {
      "filter": [ { "term": { "segment": "retail" } },
                  { "range": { "created_at": { "gte": "now-90d" } } } ]
    }
  }
}

Continual learning

Indexes can be organized by time, with rollover creating a new index per period. This makes it simple to read just the newest data, compare distributions over time, or drop old data by lifecycle rules. A scheduled job can aggregate a field by day and compare it with a reference period to detect drift. The results tell you when a model or its context set should be refreshed.

Imputation

Aggregations give the statistics that simple imputation needs: medians, percentiles, most common values per group. They answer questions such as "what is the median total for each country" in one call. Store imputed values in a separate index or field with a flag, so observed and estimated values never mix.

Anomaly detection

Elasticsearch is widely used for anomaly detection on logs and metrics. Simple rules, such as thresholds on rolling averages, percentile comparisons, or rare terms, are easy to express as aggregations. Model scores from an external tabular model can be indexed next to the original documents, and then used for alert rules, dashboards, and drill-down. Because search is fast, an analyst can jump from a flagged row to similar rows in seconds.

Sector and semantic classification

Deciding what a column means, or which sector a table belongs to, is a semantic problem. Column names, descriptions, and sample values are text, which suits a search engine. You can index column profiles and known label definitions, then retrieve the closest matches as candidates for a classifier. Retrieval narrows the choices, and a model makes the final decision. Results can be cached and indexed with a version, so repeated tables are classified instantly.

Retrieval for language-model agents

Agents that answer questions about tables often need to find the right table, the right column, or the right rows first. This is a retrieval-augmented generation setup, and an engine like this can supply the retrieval step, alone or together with a vector database. For ideas on evaluation of such setups, see /model/benchmarks.

Practical guidance and best practices

Mappings and data modeling

• Define explicit mappings for important indexes.
• Use keyword for exact values, such as ids and categories, and text for free text.
• Use the right numeric types and real date types.
• Avoid very large numbers of fields, which can bloat the cluster state.
• Denormalize at index time so queries need no joins.
• Keep the source of truth elsewhere, and make indexing repeatable from it.
• Use aliases so you can swap in a new index without changing clients.

Indexing

• Use the bulk API with moderately sized batches.
• Raise the refresh interval during large loads, then restore it.
• Use stable document ids if you need idempotent retries.
• For time-based data, use rollover and lifecycle policies.
• Reindex into a new index when you must change a mapping.

Searching

• Use filters for exact conditions, which are cached and do not affect scoring.
• Limit the fields you return with source filtering.
• Avoid wildcard queries with a leading wildcard on large fields.
• Use search-after instead of deep from and size pagination.
• Profile slow queries with the profile feature.

Cluster operations

• Keep shard sizes in a healthy range, often from several gigabytes to a few tens of gigabytes, and avoid very many tiny shards.
• Use at least one replica for important indexes.
• Separate node roles in larger clusters, such as dedicated master-eligible nodes.
• Monitor heap use, search and indexing latency, rejected requests, and disk watermarks.
• Take snapshots to a repository and test restores.
• Turn on security: authentication, role-based access, and encrypted traffic.

Preparing exports for models

• Export with a scan or point-in-time search, not deep pagination.
• Select only the fields you need.
• Convert types explicitly and record the mapping version.
• Count missing values per field and keep that report.
• Remove or hash personal data in free-text fields.
• Record the query and the time of export for reproducibility.

Sizing shards by example

• Suppose you expect 600 GB of data in an index, with one replica.
• With target shard sizes of about 30 GB, you need about 20 primary shards.
• With one replica each, the cluster stores 40 shards in total and about 1.2 TB on disk before overhead.
• If you started with 200 shards of 3 GB each, every search would fan out to 200 shards and the cluster would spend effort coordinating tiny pieces.

The numbers depend on workload, so treat them as a starting point and test. The lesson is that shard count is a design decision, not a default to accept.

Relevance tuning basics

• Boost important fields, such as titles over body text.
• Use synonyms for domain terms, such as abbreviations.
• Test with a fixed list of real queries and expected top results, and track changes over time.
• Combine text scores with business signals, such as recency, only after the basic ranking is sound.
• Review zero-result queries regularly, since they reveal vocabulary gaps.

A short launch checklist

1. Write the top five searches and aggregations.
2. Design the mapping for them.
3. Decide how data gets in, and how to rebuild the index from the source.
4. Choose shard and replica counts, and a lifecycle policy.
5. Create a read-only role for analysts and model export jobs.
6. Load test with realistic data and queries.
7. Set up snapshots and monitoring.
8. Document how to reindex safely.

Common mistakes

• Treating Elasticsearch as the only copy of important data.
• Relying on dynamic mapping and getting wrong types.
• Creating thousands of small shards.
• Mapping high-cardinality ids as text rather than keyword.
• Using deep from and size pagination to export everything.
• Updating huge documents often for small changes.
• Running heavy aggregations on text fields rather than keyword or numeric fields.
• Ignoring disk watermarks until the cluster turns read-only.
• Exposing the cluster to the internet without security.
• Exporting free text with personal data to training sets.
• Mistaking a missing field for zero when preparing features.

Frequently asked questions

What is Elasticsearch used for?

Elasticsearch is used for application and site search, log and metrics analysis, security analytics, and retrieval for language-model applications. It is chosen when you need to find documents quickly by words and filters and to summarize them with aggregations. It usually works beside a primary database.

Is Elasticsearch a database?

It stores documents and can be queried, so it behaves like a database in some ways, but it is optimized for search and analytics rather than transactions. Most teams keep a primary store for the source of truth and treat the search index as a rebuildable copy.

What is an inverted index?

It is a map from each term to the list of documents that contain it, like the index at the back of a book. With it, the engine can find documents containing certain words without scanning every document. It is the core structure behind full-text search.

What is the difference between text and keyword fields?

A text field is analyzed into tokens for full-text search, so partial matches and relevance ranking work. A keyword field stores the exact value and is used for filtering, sorting, and grouping. Many string fields are mapped as both, so you can search and aggregate on them.

How does Elasticsearch scale?

An index is split into shards that are spread across nodes, and each shard can have replicas. Adding nodes lets the cluster hold more data and serve more queries. Planning the number and size of shards early avoids trouble later, since changing it usually needs a reindex.

Can Elasticsearch do analytics?

Yes, it supports aggregations such as counts, sums, averages, percentiles, histograms, and nested group-bys. It is good for interactive exploration of log-like data. For very large scans or complex joins, a column-oriented analytical database such as ClickHouse is often faster.

How does it relate to tabular foundation models?

It helps find and prepare the data that such models read. It can filter rows, profile columns, search column descriptions, pick similar labeled rows for in-context learning, and store model scores next to the data. The models themselves treat tables as rows and columns, as described at /model/research.

Can I use it for vector search?

Yes, recent versions support dense vectors and nearest neighbor search, and you can combine it with keyword scoring for hybrid retrieval. For very large vector collections or special needs, dedicated vector databases may be a better fit, such as Weaviate or Pinecone.

Why do updates feel expensive?

Lucene segments are immutable, so an update marks the old version as deleted and writes a new one. The old data is removed later during a merge. Frequent updates to large documents therefore create extra work, and designs that append new documents or use smaller documents are often cheaper.

How should I export data from Elasticsearch for training?

Use a scan or point-in-time search with search-after, pick only the needed fields, flatten nested values, convert types explicitly, and check the row count. Also measure missing values per field and remove sensitive text. Save the query and mapping version with the export so the run can be reproduced.

Glossary

• Aggregation: A summary computed over matching documents.
• Alias: A pointer name that can be switched between indexes.
• Analyzer: The text processing chain applied to a field.
• BM25: A common text relevance scoring function.
• Doc values: Column-oriented on-disk data used for sorting and aggregations.
• Document: A JSON object stored in an index.
• Index: A named collection of documents.
• Inverted index: A term-to-documents lookup structure.
• Keyword field: A field stored as an exact value.
• Lucene: The search library that Elasticsearch is built on.
• Mapping: The definition of fields and types in an index.
• Node: One server in a cluster.
• Primary shard: The main copy of a partition of an index.
• Refresh: The step that makes new writes searchable.
• Replica shard: A copy of a primary shard.
• Rollover: Creating a new index when the current one is large or old.
• Segment: An immutable file set inside a shard.
• Text field: A field analyzed for full-text search.

Summary

Elasticsearch is a distributed search and analytics engine built on an inverted index. It supports full-text search with relevance ranking, exact filtering, aggregations, near real-time indexing, and vector search, and it scales with shards and replicas. It works best as a searchable copy of data held elsewhere, and it needs careful mapping and shard planning.

For tabular foundation models, it serves as a retrieval and profiling layer. It can select and export rows, measure missing values per column, search column descriptions when names are unclear, pick similar labeled rows as context for in-context learning, slice data by time for continual learning, supply statistics for imputation, and store anomaly scores and semantic labels next to the data. Careful mappings, repeatable exports, and clean handling of missing values keep this link reliable. To learn more about the models, see /model/research, /model/architecture, and /model/benchmarks.

Related

It belongs to Databases and sync in my stack, under Data engineering and lakehouse. Others in the same group: