Databricks

Overview

Databricks is a managed data and AI platform built around the lakehouse idea: keep data as open files in cloud storage, add a table layer that gives warehouse-like reliability, and run data engineering, SQL analytics and machine learning on the same copy. It grew out of the team behind Apache Spark and is closely tied to Delta Lake and MLflow.

For people who work with tabular data, Databricks is a place to ingest raw tables, clean them with code and SQL, train and track models, and serve results. It is also a natural place to run tabular foundation models such as TabPFN style systems on large governed tables, because sampling, feature building and scoring can all happen in one environment.

What it is

Databricks is a cloud service. You create a workspace, attach compute, and work in notebooks, SQL editors, jobs and pipelines. Data lives in your cloud storage account, in open formats. The platform provides the compute, the metadata, the governance and the user interface.

The lakehouse idea in one picture

One copy of data, several kinds of work. That is the pitch of the lakehouse.

Main parts of the platform

• Workspace: the web environment with notebooks, repos, jobs and dashboards.
• Compute: clusters for data engineering and machine learning, SQL warehouses for queries, and serverless options.
• Delta Lake tables: the default table format, with transactions and time travel.
• Unity Catalog: a central place for permissions, lineage and data discovery.
• Workflows (jobs): scheduled and triggered pipelines.
• Pipelines: declarative definitions of transformations, with quality checks.
• MLflow: tracking, packaging and registry for models.
• Model serving: hosted endpoints for models.
• Vector search and AI features: tools for retrieval and generative applications.

Feature names and availability change over time, so the official documentation is the best source for the current list.

Where it came from

Apache Spark began as a research project at a university lab in the early 2010s. It offered a faster, more flexible alternative to the earlier MapReduce style by keeping data in memory across steps and by providing a rich set of APIs. The creators of Spark then founded a company to offer it as a managed cloud service, and the platform was born.

Important steps in the story:

• Spark as the engine: batch processing, SQL, streaming and machine learning libraries on one engine.
• Notebooks in the cloud: shared, collaborative notebooks as the main interface.
• Delta Lake: introduced to add ACID transactions, schema enforcement and time travel to data lakes, later opened for wider use.
• MLflow: created to track experiments and manage the model life cycle, also released openly.
• The lakehouse concept: formalized in research and industry writing in the early 2020s as a combination of lake and warehouse properties.
• Unity Catalog and governance: unified permissions and lineage across data and AI assets.
• Generative AI features: tools for building and serving language model applications on enterprise data.

The pattern is a move from "a managed Spark" to "a general platform for data and AI".

How it works

Follow a typical job from raw files to a model. The analogy is a large kitchen with a pantry, many cooks and a recipe book.

Step 1: Data lands in cloud storage

Files arrive in object storage: CSV exports, JSON events, Parquet files, database extracts. Streams from Apache Kafka or change feeds from change data capture can also land here.

Analogy: deliveries arrive at the pantry door.

Step 2: Ingestion into tables

Ingestion tools read new files incrementally and write them into Delta tables. A common layering is called the medallion architecture:

Analogy: unpacked deliveries, then washed and chopped ingredients, then ready-to-cook kits.

Step 3: Compute is attached

A cluster is a group of machines that runs Spark. The driver plans the work, and executors do it in parallel. Clusters can scale up and down, and can be created for a job and shut down when it ends. SQL warehouses are compute tuned for queries from analysts and dashboards.

Step 4: Spark splits the work

When you run a transformation, Spark turns your code into a plan, splits the data into partitions, and sends tasks to executors. Operations that need data from other partitions, such as joins and group-bys, cause a shuffle: data is redistributed across the cluster. Shuffles are the main cost in many jobs.

Analogy: many cooks each chop their own pile of vegetables. When a recipe needs everything sorted by type, the piles must be exchanged between cooks, which takes time.

Step 5: Delta Lake keeps tables reliable

Delta tables store data as Parquet files plus a transaction log of JSON and checkpoint files. Each write adds an entry to the log. Readers use the log to see a consistent snapshot. This gives:

• ACID transactions: writes are all-or-nothing.
• Schema enforcement: bad data is rejected, and schema evolution is explicit.
• Time travel: read an older version of a table.
• Merge, update and delete: row-level changes on files.
• Optimization: compaction of small files and data layout improvements.

Step 6: Governance applies

Unity Catalog or a similar layer organizes data as catalogs, schemas and tables, and applies permissions, tags and masks. It also records lineage, that is, which tables and notebooks produced which outputs. See data lineage and data catalog.

Step 7: Models are trained and tracked

In notebooks or jobs, data scientists read Silver or Gold tables, build features, train models with common libraries, and log runs with MLflow: parameters, metrics, artifacts and the data version used. Good runs can be registered as model versions with stage labels.

Step 8: Models are deployed and monitored

A registered model can be run in batch over a table, or hosted as an endpoint. Monitoring jobs compare incoming data to training data and track quality over time.

Step 9: Orchestration ties everything together

Workflows run steps in order, with retries and alerts. Some teams use external orchestrators such as Apache Airflow to coordinate across systems.

Key concepts and terms

Open formats and portability

A key promise is that data stays in open formats in your own storage. Parquet files and Delta logs can be read by other engines. Table format interoperability with Apache Iceberg has become an important topic, since many organizations use more than one engine over the same data. Check current documentation for the level of support.

Notebooks and languages

You can write Python, SQL, Scala and R in the same notebook. This is useful for teams where analysts write SQL and engineers write Python. Code in notebooks should be moved to version-controlled repositories and tested before it becomes a production job.

A worked example

This example reads raw order files, builds a clean Delta table, creates features, trains a simple model with tracking, and queries history. Names are placeholders.

Ingest raw files into a bronze table

from pyspark.sql import functions as F

raw = (spark.read
       .option("header", True)
       .csv("/mnt/example-landing/orders/"))

bronze = (raw
          .withColumn("_ingested_at", F.current_timestamp())
          .withColumn("_source_file", F.input_file_name()))

(bronze.write
       .format("delta")
       .mode("append")
       .saveAsTable("demo.bronze.orders"))

Keeping the file name and load time makes later debugging much easier.

Clean into a silver table

bronze = spark.table("demo.bronze.orders")

silver = (bronze
    .select(
        F.col("order_id").cast("long"),
        F.col("customer_id").cast("long"),
        F.to_date("order_date").alias("order_date"),
        F.when(F.col("channel").isin("", "N/A"), None)
         .otherwise(F.col("channel")).alias("channel"),
        F.col("amount").cast("decimal(12,2)"),
        (F.col("returned") == "1").cast("int").alias("returned"),
    )
    .dropDuplicates(["order_id"]))

silver.write.format("delta").mode("overwrite").saveAsTable("demo.silver.orders")

Notice that placeholders like an empty string or "N/A" become real nulls, and amounts get an exact decimal type.

The same cleanup with SQL

CREATE OR REPLACE TABLE demo.silver.orders AS
SELECT CAST(order_id AS BIGINT)         AS order_id,
       CAST(customer_id AS BIGINT)      AS customer_id,
       TO_DATE(order_date)              AS order_date,
       NULLIF(NULLIF(channel, ''), 'N/A') AS channel,
       CAST(amount AS DECIMAL(12,2))    AS amount,
       CAST(returned AS INT)            AS returned
FROM   demo.bronze.orders;

Upsert changes with merge

MERGE INTO demo.silver.orders AS t
USING demo.bronze.orders_changes AS s
ON t.order_id = s.order_id
WHEN MATCHED AND s.op = 'd' THEN DELETE
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED AND s.op <> 'd' THEN INSERT *;

This is how a stream from change data capture is applied to a lake table.

Build features

from pyspark.sql import Window

w = Window.partitionBy("customer_id").orderBy("order_date")

features = (spark.table("demo.silver.orders")
    .withColumn("prev_amount", F.lag("amount").over(w))
    .withColumn("days_since_prev",
                F.datediff("order_date", F.lag("order_date").over(w)))
    .select("order_id", "channel", "amount",
            "prev_amount", "days_since_prev", "returned"))

features.write.format("delta").mode("overwrite").saveAsTable("demo.gold.order_features")

The first order of each customer has null lag values. That is true missingness and should stay as such.

Train and track a model

import mlflow
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

pdf = spark.table("demo.gold.order_features").toPandas()
pdf["channel"] = pdf["channel"].astype("category")

X = pdf.drop(columns=["returned"])
y = pdf["returned"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)

with mlflow.start_run(run_name="returns-baseline"):
    model = HistGradientBoostingClassifier(categorical_features="from_dtype")
    model.fit(X_tr, y_tr)
    auc = roc_auc_score(y_te, model.predict_proba(X_te)[:, 1])
    mlflow.log_param("model", "hist_gbm")
    mlflow.log_metric("auc", auc)
    mlflow.log_param("source_table_version",
                     spark.sql("DESCRIBE HISTORY demo.gold.order_features LIMIT 1")
                          .collect()[0]["version"])

Logging the table version means you can later reread exactly the data used for this run.

Query an older version

SELECT COUNT(*) FROM demo.gold.order_features VERSION AS OF 3;
SELECT COUNT(*) FROM demo.gold.order_features TIMESTAMP AS OF '2025-03-01 00:00:00';

A small streaming sketch

stream = (spark.readStream
          .format("cloudFiles")
          .option("cloudFiles.format", "json")
          .load("/mnt/example-landing/events/"))

(stream.writeStream
       .option("checkpointLocation", "/mnt/example-checkpoints/events")
       .trigger(availableNow=True)
       .toTable("demo.bronze.events"))

The checkpoint location remembers progress, so a restart continues where it stopped without duplicates.

A baseline to compare with

A useful habit: before building something complex, compute a simple baseline (a majority class, or a plain gradient boosting model) and log it. Later models, including foundation models used with in-context learning, can then be judged against a shared benchmark on the same split. See the benchmarks page for how such comparisons are framed.

Strengths and limitations

Strengths

• One platform for many jobs: engineering, SQL and ML on shared data.
• Open formats: data stays in open files in your own storage.
• Scale: Spark handles very large data through distributed processing.
• Reliable tables: Delta Lake adds transactions, time travel and merge.
• Collaboration: shared notebooks and repositories.
• ML life cycle tools: tracking, registry and serving.
• Governance: central permissions and lineage.
• Streaming and batch: one API style for both.
• Flexible compute: clusters sized for the job.

Limitations

• Complexity: many features, many choices, and a real learning curve.
• Cost control: clusters left running or oversized add up. Monitor and set policies.
• Distributed overhead: for small data, a single machine library is simpler and faster.
• Shuffle-heavy jobs: poorly designed joins and skew can be slow.
• Notebook habits: untested notebook code can become fragile production code.
• Platform coupling: some features are specific to the platform even if data formats are open.
• SQL-first teams: pure warehouse users may find a platform like Snowflake simpler for plain SQL analytics.

How it compares with alternatives

Platform versus engine

Apache Spark is the engine. Databricks is a platform that includes Spark plus governance, jobs, notebooks and extras. You can run Spark without the platform, for example on your own cluster, but then you handle upgrades, scaling, security and monitoring yourself.

Table formats

Delta Lake and Apache Iceberg both add table features to files. Iceberg also has topics such as Iceberg cataloging and Iceberg compaction. Which to use depends on your engines and ecosystem.

Stream processing options

Beyond Spark's structured streaming, engines like Apache Flink are also common for low-latency stream work. See Spark Streaming for the Spark approach.

Role in tabular foundation models

Tabular foundation models learn from tables: rows, typed columns, missing cells and often anonymous columns. Databricks is not itself such a model, and no specific integration with any model is claimed here. It is a platform where the tables live and where models can be run, tracked and governed. Below is how its features relate to the needs of these models.

Row and column structure

Delta tables are columnar and typed, which suits tabular input. Spark can join, aggregate and window across many tables to flatten relational data into the single wide table that a tabular model reads. For small context samples, the final step usually converts a Spark data frame to a pandas data frame on one machine, because in-context models typically read a limited number of rows.

Mixed types

Spark has strong typing, with integers, decimals, strings, dates, timestamps, arrays and structs. Before modeling, flatten nested structures, convert date columns into useful numbers (such as days since an event), and cast codes to categories deliberately. Decimal precision should be preserved during storage and cast carefully for the model.

Missing values

Spark nulls map naturally to the missing cells a model can handle. Silver-layer cleaning should convert placeholders such as empty strings or sentinel numbers to nulls, and record which columns had placeholders. For filling genuine gaps, see imputation networks. A good rule is to keep both the original and the imputed column.

No column names

Pipelines sometimes produce generic names after unions or pivots. Models that do not depend on names still need a stable column order and types. Unity Catalog comments, tags and the data catalog hold human context. Keep a mapping from generic names to meaning, so results can be explained to people.

Pre-training on synthetic or real tables

Large-scale pre-training of tabular foundation models commonly uses generated tables, see synthetic data generation. Databricks and Spark can generate such data at scale in parallel, because each table is an independent task. For real tables, the platform's governance features help with access control and lineage when using sensitive data. The research page describes how pre-training differs from the use stage.

In-context learning

In-context models take a labeled sample and the rows to predict in one pass. On the platform, this means: build a Gold table, sample it in a repeatable way (hash of the key), stratify when classes are rare, convert to pandas, and call the model. Because Delta tables have versions, the exact context used in a past prediction can be rebuilt. MLflow can log the table version and sample parameters with every run.

Continual learning

New data arrives continually. Structured streaming and incremental ingestion keep tables fresh, and merge operations apply changes. Comparing the distribution of today's data with the data used for the last context refresh shows when an update is due. Time travel makes before-and-after comparisons simple. See schema drift detection and schema evolution.

Imputation

Spark can run imputation across very large tables, with simple statistics or model-based methods. Store the results in separate columns, with a flag, in a Gold table. Use Delta versions to compare the effect of different imputation strategies on a downstream task.

Anomaly detection

Large-scale scanning is a strength. Rules, statistics and model scores can run over all rows, with flagged records written to a table that a review team uses. See anomaly detection. Streaming jobs can score events as they arrive and send alerts.

Sector and semantic classification

Text and code fields can be classified in batch with a model applied through a user-defined function; see Spark UDF de-identification for a related pattern that applies custom logic per row. Results go to a table with the key, label, confidence and model version, so downstream users can filter by confidence and rerun when the model improves.

Governance and privacy

Sensitive fields can be masked or tokenized before modeling; see data tokenization and data contract validation. Lineage shows which tables fed which models. Model architecture and evaluation background is on the architecture page.

Practical guidance and best practices

Architecture

• Use the medallion layers: bronze for raw, silver for clean, gold for use.
• Keep raw data immutable so you can always reprocess.
• Use one table per entity or event type, with clear keys and timestamps.
• Register tables in a catalog with owners and descriptions.

Performance

• Right-size clusters. Start small and scale if needed.
• Use auto-termination. Idle clusters should stop.
• Avoid unnecessary shuffles. Filter early, select only needed columns, and broadcast small tables in joins.
• Watch for skew. One huge key can make one task slow.
• Compact small files. Many tiny files slow reads.
• Use columnar formats. Do not store analytics data as CSV.
• Avoid collecting big data to the driver. Bring only samples to one machine.

Reliability

• Make jobs idempotent so retries are safe.
• Use checkpoints for streaming jobs.
• Add data quality checks at the silver boundary.
• Alert on failures and on unexpected row count changes.
• Use time travel and table history for recovery.

Machine learning

• Track every run with MLflow, including data version and parameters.
• Build features once in Gold tables and reuse them.
• Keep training, validation and test splits fixed and recorded.
• Prevent leakage: compute features as of the label date.
• Compare against a simple baseline, such as a tuned gradient boosting model.
• Monitor input distributions after deployment.

Code quality

• Keep code in version control and use tests.
• Parameterize notebooks and jobs; avoid hard-coded paths.
• Separate transformation logic from orchestration.
• Use dbt or a similar tool for SQL models if your team prefers that style.

Security

• Use least-privilege access and group-based permissions.
• Store secrets in a secret store, never in notebooks.
• Mask personal data and limit who can see raw layers.
• Review audit logs.

Common mistakes

1. Using a cluster for tiny data. Distributed overhead outweighs the benefit. Use a single machine for small samples.
2. Leaving clusters running. Set auto-termination and policies.
3. Collecting a huge data frame to the driver. This crashes or stalls the notebook.
4. Ignoring data skew. One hot key makes a job crawl.
5. Too many small files. Compact tables and size writes sensibly.
6. Overwriting tables without versions in mind. Know the retention of history before relying on time travel.
7. Notebook-only production. Move code to tested, version-controlled jobs.
8. Not logging the data version. Without it, results cannot be reproduced.
9. Placeholders as real values. Convert "N/A", empty strings and sentinels to nulls.
10. Leaking labels into features. Use only information known at prediction time.
11. No quality gate between layers. Bad data flows quietly into gold tables.
12. Skipping governance setup. Retrofitting permissions and lineage later is painful.

Frequently asked questions

What is Databricks used for?

Databricks is used to ingest and transform large amounts of data, run SQL analytics, train and track machine learning models, and serve results. It is popular with teams that need data engineering and data science on the same governed data, and with organizations that want to keep data in open formats in their own cloud storage.

What is a lakehouse?

A lakehouse is an architecture that stores data as open files in cheap object storage like a data lake, but adds a table layer that provides transactions, schema control, time travel and fast SQL like a data warehouse. The goal is one copy of data that serves reporting, engineering and machine learning.

How is Databricks related to Apache Spark?

Spark is the open-source distributed processing engine at the heart of the platform. Databricks was founded by its creators and provides a managed service around it, adding notebooks, job scheduling, governance, optimized runtimes and machine learning tools. You can run Spark without Databricks, but you must then manage the infrastructure yourself.

What is Delta Lake?

Delta Lake is a table format that stores data as Parquet files and keeps a transaction log beside them. It adds atomic writes, schema enforcement, time travel and row-level merge, update and delete to data lake files. It makes lake tables reliable enough for production reporting and pipelines.

What is the medallion architecture?

It is a layering pattern: bronze tables hold raw data as received, silver tables hold cleaned and deduplicated data, and gold tables hold business-ready aggregates and features. Each step increases quality and trust, and keeping the raw layer lets you reprocess when logic changes.

Is Databricks a data warehouse?

It can serve warehouse-like workloads through SQL warehouses and Delta tables, but its roots and strengths are broader: data engineering, streaming and machine learning on lake data. Pure SQL analytics teams sometimes prefer a dedicated warehouse, while teams with mixed workloads value having a single platform.

How does MLflow fit in?

MLflow records experiments, including parameters, metrics, code versions and artifacts, and manages model versions in a registry. On the platform it is integrated, so each run can link to the table version it used. That gives reproducibility and an audit trail from data to deployed model.

Do I need Spark for small data?

Usually not. If the data fits comfortably in memory on one machine, ordinary data frame libraries are simpler and often faster. Spark pays off when data or computation is too large for a single machine, or when you need to run the same job across many partitions in parallel.

Can Databricks be used with change data capture?

Yes. Change events from a source database can land as files or arrive through a stream, and merge statements apply inserts, updates and deletes to Delta tables. Remember to handle schema changes and to reconcile counts against the source regularly.

How does Databricks relate to tabular foundation models?

It is a convenient environment for preparing the tables such models read, sampling repeatable context sets, tracking runs and storing predictions. The models themselves are separate pieces of software. For details on how they read tables, see the model research page.

Glossary

• Auto-termination: automatic shutdown of idle compute.
• Broadcast join: sending a small table to all workers to avoid a shuffle.
• Bronze / silver / gold: layers of raw, clean and curated data.
• Cluster: machines used together to run Spark jobs.
• Delta Lake: a table format with a transaction log.
• Driver: the process that coordinates a Spark job.
• Executor: a worker process that runs tasks.
• Feature table: a table of reusable model inputs.
• Lakehouse: lake storage with warehouse-like table features.
• Lineage: a record of how data flows between assets.
• MLflow: tool for tracking experiments and managing models.
• Notebook: interactive document mixing code and results.
• Partition: a slice of data processed by one task.
• Shuffle: moving data between workers.
• Skew: uneven distribution of data across partitions.
• Time travel: reading an earlier version of a table.
• Unity Catalog: central governance layer for data and AI assets.

Summary

Databricks is a managed lakehouse platform built on Apache Spark, Delta Lake and MLflow. It lets teams keep data in open formats, process it at scale, run SQL, train and track models, and govern everything centrally. Its strengths are breadth and scale, and its main risks are complexity and cost discipline.

For tabular foundation models, it provides the tables, repeatable sampling, versioned snapshots, scoring at scale and governance around them. Pair it with sound cleaning, honest missing values and careful leakage control. Related reading includes Snowflake, data synchronization and the model benchmarks page.

Related

It belongs to Databases and sync in my stack, under Data engineering and lakehouse. Others in the same group: