MongoDB

Overview

MongoDB is a document database. Instead of rows in fixed tables, it stores documents: JSON-like records that can hold nested objects and lists. Documents live in collections, and two documents in one collection do not have to share the same fields. This makes it a natural fit for applications whose data shape changes often or is deeply nested.

It is a general purpose operational database. It offers indexes, rich queries, an aggregation pipeline, replication for availability, and sharding for scale. It is less suited to heavy columnar analytics than a column store, and its flexibility needs discipline to avoid messy data. This page explains how it works and how it relates to tabular foundation models, which learn from flat tables of rows and columns. The key bridge is flattening: turning nested documents into clean tables that a model can read. For the model side, see /model/research.

What it is

MongoDB is a database server that stores data as BSON, a binary form of JSON that adds types such as dates, 64-bit integers, decimals, and binary data. You work with it through drivers for many programming languages, a shell, and a query language based on JSON-style filters.

Main traits:

• Documents, not rows. One document holds an entity and its nested parts.
• Collections, not tables. A collection groups similar documents.
• Flexible schema. Fields can differ between documents, and optional validation rules can add structure.
• Rich indexes. Single field, compound, multikey for arrays, text, geospatial, and TTL indexes.
• Aggregation pipeline. A staged way to filter, group, join, reshape, and compute.
• Replica sets. A primary and secondaries keep copies of the data, with automatic failover.
• Sharding. Collections can be split across many servers by a shard key.
• Transactions. Multi-document transactions exist for cases that need them.

Everyday analogy

A relational table is like a printed form where every person fills the same boxes. A document database is like a folder per person. Each folder holds what is relevant: one has a passport copy, another has three letters and a photo. You can still search all folders by name or by date, but you do not need to design one form that fits everybody.

Where it came from

MongoDB appeared in the late 2000s, during the wave of NoSQL systems built for web applications that needed to scale out and change quickly. Developers were tired of mapping application objects to many joined tables. A document that mirrors the object in code seemed simpler. The name comes from the word "humongous".

Since then it has added many features that early NoSQL systems lacked: a more capable storage engine, multi-document transactions, schema validation, time series collections, change streams, and search and vector features in its managed offering. The company behind it also runs a cloud service. As with any fast-moving product, check current documentation for exact feature status.

The broader idea of document stores goes back further, to systems for semi-structured data and to the long use of JSON as an exchange format on the web.

How it works

Step 1: A document is stored

A document is a set of field and value pairs. Each document has a unique _id field. Values may be strings, numbers, dates, arrays, or other documents.

{
  "_id": "o-1001",
  "customer": { "id": "c-77", "country": "DE" },
  "items": [
    { "sku": "A1", "qty": 2, "price": 9.5 },
    { "sku": "B7", "qty": 1, "price": 20.0 }
  ],
  "status": "paid",
  "created_at": "2026-03-04T10:15:00Z"
}

In a relational design, this order would span an orders table, a customers table, and an order items table. Here it is one document, read in one operation.

Step 2: Storage on disk

The default storage engine stores documents in compressed pages organized by a tree structure and writes changes to a journal first for durability. It uses document-level concurrency control, so many writers can work on different documents at once. Think of it as a filing cabinet where each folder can be locked on its own while others stay open.

Step 3: Indexes find documents quickly

Without an index, a query scans the whole collection. With an index, the server walks a sorted tree to the matching entries. A compound index on (customer.country, created_at) supports queries that filter by country and sort by time. Indexes cost memory and write time, so choose them with care.

Step 4: Queries and the aggregation pipeline

Simple queries use a filter document.

db.orders.find(
  { status: "paid", "customer.country": "DE" },
  { _id: 1, created_at: 1 }
).sort({ created_at: -1 }).limit(20)

For more complex work, the aggregation pipeline passes documents through stages, like items on a conveyor belt: match, group, sort, project, unwind, lookup, and more.

db.orders.aggregate([
  { $match: { status: "paid" } },
  { $unwind: "$items" },
  { $group: {
      _id: "$customer.country",
      revenue: { $sum: { $multiply: ["$items.qty", "$items.price"] } },
      orders: { $addToSet: "$_id" }
  }},
  { $project: { revenue: 1, order_count: { $size: "$orders" } } },
  { $sort: { revenue: -1 } }
])

Step 5: Replication

A replica set has one primary that takes writes and secondaries that copy its operation log. If the primary fails, the members hold an election and one secondary becomes primary. Write concern controls how many members must confirm a write. Read preference controls where reads go. A higher write concern is safer but slower.

Step 6: Sharding

When one server is not enough, a collection is sharded by a shard key. The data is split into chunks by key range or by hash, and spread over shards. A router sends each query to the right shard or to all shards. A good shard key spreads load evenly and appears in common queries. A poor one creates a hot shard.

Step 7: Change streams

Applications can subscribe to changes in a collection. This is useful for keeping a search index, a cache, or an analytics store in sync with the operational data.

Key concepts and terms

A worked example

We will store customer orders, query them, and produce a flat table for a tabular model.

Insert and query

from pymongo import MongoClient

client = MongoClient("mongodb://reader:placeholder@example.internal:27017/shop")
db = client["shop"]

db.orders.insert_one({
    "_id": "o-1002",
    "customer": {"id": "c-78", "country": "FR"},
    "items": [{"sku": "A1", "qty": 1, "price": 9.5}],
    "status": "paid",
    "created_at": "2026-03-05T08:00:00Z",
})

db.orders.create_index([("customer.country", 1), ("created_at", -1)])

for doc in db.orders.find({"status": "paid"}).limit(3):
    print(doc["_id"], doc["customer"]["country"])

Flatten into a table

A tabular model expects each row to have the same columns. The pipeline below turns each order into one row with numeric and categorical columns.

db.orders.aggregate([
  { $project: {
      _id: 0,
      order_id: "$_id",
      country: "$customer.country",
      status: 1,
      n_items: { $size: "$items" },
      total: { $sum: {
        $map: { input: "$items", as: "i",
                in: { $multiply: ["$$i.qty", "$$i.price"] } } } },
      created_at: 1
  }}
])

In Python, load the result into a data frame.

import pandas as pd

rows = list(db.orders.aggregate([
    {"$project": {"_id": 0, "order_id": "$_id",
                  "country": "$customer.country",
                  "status": 1,
                  "n_items": {"$size": "$items"}}}
]))
df = pd.DataFrame(rows)
print(df.dtypes)
print(df.isna().mean())

The last line prints the share of missing values per column. Because documents may lack fields, you will often find nulls that are not true unknowns but simply fields that were never written. Decide how to treat them before training.

What to check in the flattened table

1. Do all rows have the same columns? If not, the missing ones should be explicit nulls.
2. Are numbers stored as numbers? Mixed types in one field, such as "12" and 12, will break a column.
3. Are dates real dates? A model needs a consistent format.
4. Are arrays reduced to counts, sums, or other summaries, or exploded into child tables?
5. Is there a stable key, so you can join predictions back?

Embedding versus referencing in practice

Take a blog with posts and comments. Two designs are common.

Design A embeds comments inside each post document.

{ "_id": "p1", "title": "Hello", "comments": [ { "by": "u1", "text": "Nice" } ] }

Design B keeps comments in their own collection and points to the post.

{ "_id": "c1", "post_id": "p1", "by": "u1", "text": "Nice" }

The trade-offs:

A good rule of thumb: embed when the child data is small, bounded, and always read with its parent. Reference when it is large, grows without limit, or is queried on its own.

Modeling for a tabular use case

If you know that a model will later read this data, a few small habits help right away.

• Write timestamps as real date values, not strings in mixed formats.
• Keep money as a decimal type or as integer minor units, not as floating text.
• Store category fields as short, stable codes and keep the labels elsewhere.
• Keep a schema_version field in each document, so that old and new shapes can be told apart.
• Write explicit nulls for known but unknown values, and leave out fields that do not apply.

Profiling a collection

A quick profile tells you how messy a collection is. This pipeline counts the types seen for one field.

db.orders.aggregate([
  { $group: { _id: { $type: "$customer.country" }, n: { $sum: 1 } } }
])

If the output shows both string and null, plus a count for missing, you know the cleaning step has to handle three cases. Run similar checks for each key field and write the results down. This is the document-store equivalent of a column profile.

Strengths and limitations

Strengths

• Flexible schema that follows the application objects.
• One read can return a whole entity with its nested parts.
• Easy to start and iterate on, with good driver support.
• Rich indexing, including arrays, geospatial, and text.
• Built-in replication with automatic failover.
• Horizontal scale through sharding.
• Aggregation pipeline for in-database transformations.
• Change streams for event-driven designs.

Limitations

• Flexible schema can become chaos without validation and conventions.
• Joins are possible with lookup, but they are not its strength, and heavy join workloads fit relational systems better.
• Large analytical scans are slower than in a column store.
• Documents have a size limit, so unbounded arrays inside one document are a trap.
• Poor shard key choice is hard to fix later.
• Memory sizing matters, since indexes and the working set should fit in RAM for best speed.
• Transactions spanning many documents or shards cost more than single-document operations.

How it compares with alternatives

A very common architecture keeps MongoDB as the operational store and copies data to an analytical store for reporting and model training. Change streams or a connector such as Kafka Connect can feed that copy.

Role in tabular foundation models

A tabular foundation model learns from tables: each row is a record, each column a typed feature, and the model must cope with mixed types, missing values, columns without useful names, and unknown relationships. A document database holds data that is not yet in that shape. Its role is therefore the source system that must be flattened, cleaned, and profiled before a model can use it. See /model/architecture for how a model reads a table.

Row and column structure

Documents are nested and may be ragged. A model needs rectangular input. The first job is to decide what a row means. Is it an order, a customer, or an order line? Each choice gives a different table. The aggregation pipeline can produce any of them with unwind and project stages.

Mixed types

A field in a schemaless collection can hold different types over time, such as a number in old documents and a string in newer ones. Before sending data to a model, normalize types. Schema validation and type checks in the pipeline help, and a quick profile that lists the types seen per field is a good first step.

Missing values

In document data, a missing field and a null value are different things, and a third case, an empty string, is common too. All three often mean "unknown", but not always. A cleaning step should map them to one explicit missing marker and record which kind was seen if that matters. Tabular models handle missing cells better when the marker is consistent. Missing values may also be informative: a field that is absent for a customer segment can itself be a signal.

No column names

After flattening, field paths like customer.country or items.0.price become column names. These are usually meaningful, but legacy collections may use cryptic short keys to save space. A model that does not depend on names can still work, but people need a data dictionary to understand results. Keep a mapping from cryptic keys to meanings.

Pre-training data

Tabular foundation models are pre-trained mostly on synthetic tables, sometimes with real ones. Document stores are a rich source of real, messy data, but you must flatten and anonymize first. Sampling can be done with a pipeline stage that draws a random subset, which helps when building varied training collections.

db.orders.aggregate([{ $sample: { size: 100000 } }])

In-context learning

In in-context learning, a model reads labeled example rows and predicts for new rows without weight updates. The context must be small and well chosen. A document database can retrieve these examples: the most recent paid orders, a balanced sample by label, or similar customers selected through an index. The flattened examples are then passed to the model as a table.

Continual learning

Operational data changes every day. A change stream or a timestamp field lets you pull only new documents since the last run, which supports incremental updates and drift checks. Keep a field such as updated_at indexed, and record the last processed value.

Imputation

Imputation fills missing cells. In a document store, you may be tempted to write imputed values back into the source documents. Avoid that. Write them to a separate collection or table with a flag, so that real observations and estimates never mix. The pipeline can also find which fields are most often absent, which guides the choice of method.

Anomaly detection

Document data often contains anomalies: malformed records, impossible dates, duplicate ids, or sudden spikes. Pipeline stages can count by field and day and flag outliers. A model can score the flattened rows, and the scores can be stored in a new collection that links to the original _id.

Sector and semantic classification

Deciding what a field means, or which sector a record belongs to, is a semantic task. For documents, evidence includes field names, value patterns, and nested structure. Computing a compact profile per field, such as type counts, distinct counts, and sample values, gives a classifier rich input without exposing all raw data. Embedded structure, such as an address object, is itself a strong clue to meaning.

Vectors and retrieval

Modern document databases can store vectors next to documents, which helps retrieval for models. Treat this as an optional extra, and compare it with dedicated systems such as Qdrant or Milvus when scale or latency needs are high. See also hybrid search for combining keyword and vector retrieval. For benchmark context, see /model/benchmarks.

Practical guidance and best practices

Schema design

• Design around access patterns. Ask what the application reads together, and embed it.
• Embed data that is read together and has a bounded size, such as order lines.
• Reference data that grows without bound, such as a user's full event history.
• Avoid very large arrays inside a document.
• Use consistent field names and types across documents.
• Add schema validation for fields that must exist.
• Use a stable, meaningful key for lookups, and avoid putting large values in _id.

Indexing

• Create compound indexes that match your common filter and sort together.
• Follow the rule of equality fields first, then sort fields, then range fields.
• Drop unused indexes, since each one slows writes.
• Use partial indexes for a subset of documents.
• Use TTL indexes to expire temporary data.
• Check queries with the explain feature and look for collection scans.

Operations

• Run a replica set in production, never a lone server.
• Choose write concern to match how much data loss you can tolerate.
• Monitor replication lag, cache usage, slow queries, and disk.
• Take backups and test restores.
• Secure the server with authentication, network limits, and encryption in transit and at rest.
• Plan the shard key carefully before sharding.

Preparing data for models

• Fix the grain of the table first: one row per what?
• Flatten nested objects with the pipeline, not in application code that differs from run to run.
• Summarize arrays into counts, sums, min, max, or last value, or explode them into a child table.
• Convert types explicitly.
• Remove or hash direct identifiers.
• Cut data at a time boundary for evaluation.
• Version the pipeline so each model run can be reproduced.

A reusable flattening recipe

1. Pick the entity for the row.
2. Project scalar fields as columns.
3. Convert nested objects with dotted paths into columns.
4. Replace arrays with aggregates.
5. Add counts of missing fields as extra columns if useful.
6. Export to a columnar file such as Parquet for the model.
7. Record the pipeline text with the model version.

Choosing between MongoDB and a relational database

Ask these questions in order.

1. Is the data naturally nested and read as one unit? If yes, lean to documents.
2. Do you need many joins and ad hoc queries across entities? If yes, lean to relational.
3. Does the shape of the data change often? If yes, lean to documents.
4. Do you need strict constraints across tables? If yes, lean to relational.
5. Will write volume outgrow one server? If yes, both can scale, but check the shard design early.
6. Does your team already know one of them well? Skills often decide the outcome.

Many products use both. That is fine, as long as each has a clear job.

Handling schema evolution

• Add new fields as optional and write code that tolerates their absence.
• Backfill old documents in small batches during low traffic.
• Use a schema_version field and migrate lazily on read when volume is large.
• Keep validation in "warn" mode first, then move to "error" after cleanup.
• Never rename a field in place without a plan for readers that still use the old name.

Security checklist

• Turn on authentication and use roles with least privilege.
• Limit network access to known hosts.
• Encrypt traffic and data at rest.
• Do not log full documents that contain personal data.
• Rotate credentials and keep them in a secret store.
• Use a separate read-only role for analytics and model export jobs.

Backup and recovery thinking

• Decide how much data you can afford to lose and how long you can be down.
• Use snapshots or backup tools on a schedule, and keep copies in another place.
• Test a restore on a clean machine at least once.
• For sharded clusters, make sure backups are consistent across shards.

Common mistakes

• Treating the flexible schema as no schema at all.
• Embedding unbounded lists that grow until the document hits its size limit.
• Creating too many indexes, or none at all.
• Choosing a monotonically increasing shard key that creates a hot shard.
• Running heavy analytical scans on the primary during peak hours.
• Using a join-like lookup everywhere instead of modeling data for reads.
• Forgetting that missing, null, and empty are different.
• Training a model on documents where the same field has several types.
• Writing imputed values over source data.
• Leaking future information into features by not cutting at a time boundary.
• Exposing the database to the open internet without authentication.

Frequently asked questions

What is MongoDB used for?

MongoDB is used as an operational database for applications that handle nested or changing data, such as catalogs, user profiles, content, and event records. It is popular where developers want data in the database to look like objects in their code. It can also back real-time features through change streams.

What is the difference between a document database and a relational database?

A relational database stores rows in tables with a fixed schema and links them by joins. A document database stores whole records as nested documents with a flexible schema. Relational systems are strong for transactions and joins, while document systems are strong for evolving, hierarchical data read as one unit.

Does MongoDB have a schema?

It does not require one, but you can add one. Schema validation lets you declare required fields and types, and many teams enforce this to keep data consistent. A common good practice is to start flexible during early design and tighten the rules as the model stabilizes.

Can MongoDB handle transactions?

Yes. It supports atomic operations on a single document by default and multi-document transactions when needed. Transactions that span many documents or shards cost more, so many designs try to keep related data inside one document to avoid them.

How does MongoDB scale?

It scales reads by adding secondary members to a replica set, and it scales writes and storage by sharding a collection across multiple servers. The choice of shard key decides how evenly the load spreads, so it deserves careful thought.

Is MongoDB good for analytics?

It can run aggregations and is fine for moderate analytical needs on operational data. For very large scans, many teams copy the data to a column-oriented store such as ClickHouse or a cloud warehouse, which are built for that work.

How do I prepare MongoDB data for a tabular model?

Choose what one row means, flatten nested fields into columns, summarize arrays, normalize types, mark missing values consistently, remove direct identifiers, and export to a typed table. Then profile the result for missing rates and outliers before giving it to a model.

Can a tabular foundation model read documents directly?

Tabular foundation models expect rows and columns, so nested documents must be flattened first. Once flattened, a pre-trained model can often make predictions in context without extra training. See /model/research for how such models are built and evaluated.

Should I store imputed values back in the database?

It is safer not to overwrite source documents. Store estimates in a separate collection or table with a flag and a reference to the original id. That way you can always tell observed values from estimated ones, and you can rerun the imputation with a better method later.

How does it fit with search and vector tools?

It often works alongside them. A search engine such as Elasticsearch handles full-text search, and a vector store handles similarity search, while MongoDB remains the system of record. Change streams or connectors keep them updated.

Glossary

• Aggregation pipeline: A sequence of stages that transform documents.
• BSON: Binary JSON with extra data types.
• Change stream: A live feed of changes in a collection.
• Collection: A group of documents.
• Compound index: An index on several fields.
• Document: A JSON-like record.
• Embedding: Storing related data inside a document.
• Failover: Automatic switch to another member when the primary fails.
• Flattening: Turning nested data into rows and columns.
• Multikey index: An index over array elements.
• Primary: The replica set member that accepts writes.
• Projection: Choosing which fields a query returns.
• Read preference: A rule that picks which member serves reads.
• Replica set: A group of servers with the same data.
• Secondary: A member that copies the primary.
• Shard key: The field that decides where a document lives.
• TTL index: An index that deletes documents after a time.
• Write concern: The level of confirmation required for a write.

Summary

MongoDB is a document database that stores flexible, nested, JSON-like records in collections. It offers indexes, an aggregation pipeline, replica sets for availability, sharding for scale, and change streams for event-driven designs. It is a strong operational store for evolving data and a weaker choice for heavy columnar analytics.

For tabular foundation models, MongoDB is mostly a source system. The work lies in deciding the row grain, flattening nested fields, normalizing mixed types, treating missing values consistently, sampling and slicing by time, and storing imputed values and anomaly scores separately from observed data. Done well, this turns messy documents into the clean rectangular tables that such models read in context. To learn more about the models themselves, see /model/research, /model/architecture, and /model/benchmarks.

Related

It belongs to Databases and sync in my stack, under Data engineering and lakehouse. Others in the same group: