AWS Glue Data Catalog

Overview

The AWS Glue Data Catalog is a managed metadata repository. It stores the names, schemas, locations and partitions of tables, so that engines can find and read data by name instead of by file path. It does not hold the data itself. The data stays in places such as Amazon S3, and the catalog records where it is and what it looks like.

Many AWS analytics engines read the same catalog, including Amazon Athena, Amazon EMR and EMR on EKS. That shared view is the main value: one definition of a table, used by many tools. For teams that train or run models on raw tables, such as the tabular foundation models described on /model/research, the catalog is the map that tells a pipeline which tables exist, what columns they have and where the files are.

What it is

A data catalog is a structured index of data assets. Think of a library card catalog. The cards do not contain the books. They tell you the title, the author, the shelf and a short description. The Glue Data Catalog does the same for tables.

The main objects are few.

• Database: a logical group of tables, like a folder or schema.
• Table: a definition with a name, columns, types, a location and a format.
• Partition: a slice of a table, often by date, with its own location.
• Connection: stored information on how to reach an external data source.
• Schema registry (related): a place to keep and validate schemas for streaming data.
• Column statistics: optional numbers such as distinct count and null count.

Hive metastore compatibility

The catalog is compatible with the Apache Hive metastore API. That means tools built for Hive and Spark can use it as their metastore without running a separate database. This is a practical benefit for teams that migrate from self-managed clusters.

Catalog is not the data

This distinction is important. If you delete a table definition in the catalog, you normally remove only the metadata. The files stay where they were, unless you choose to delete them. If you delete files but leave the table definition, queries will fail or return nothing. The catalog and the storage must be kept in step.

How it relates to AWS Glue the service

AWS Glue is a broader service that includes jobs for extract, transform and load work, crawlers, and a catalog. The Data Catalog is the shared metadata part. You can use the catalog without running a single Glue job, for example when Athena queries tables you defined by hand.

Where it came from

Metadata catalogs grew out of the Hive project in the Hadoop era. Hive needed to know which files made up a table and what the columns meant, so it kept this in a metastore, usually a relational database. Other engines such as Spark and Presto adopted the same metastore, because it was a common source of truth.

Running a metastore yourself has costs: a database to maintain, backups, upgrades and scaling. Managed catalogs appeared so that teams could avoid that work. The Glue Data Catalog is the AWS version, built to be Hive compatible and shared across services.

Later, table formats such as Apache Iceberg and Delta Lake changed what a catalog must track. Instead of a list of partitions, a table now has a pointer to a metadata file that describes snapshots. Catalogs adapted by storing that pointer. The Iceberg cataloging page explains this shift in detail.

Governance also moved into the catalog's neighborhood. Permission services like AWS Lake Formation apply access control on top of catalog objects, so the catalog became the anchor for both discovery and security.

How it works

Step 1: you define a database

A database is just a name and an optional location. It groups tables. A common pattern is one database per zone or domain, such as raw_sales and curated_sales.

Analogy: creating a shelf label in the library.

Step 2: you define a table

A table entry holds:

• The table name and database.
• The list of columns with names and types.
• The partition keys, if any.
• The storage location, such as an S3 prefix.
• The input and output format and the serialization library.
• Table parameters, such as the file format, compression or table type.

You can create this by SQL DDL in Athena, through an API, through infrastructure code, or by letting a crawler infer it.

Step 3: crawlers can infer the schema

A crawler scans a data location, samples files, infers the schema and writes or updates table definitions. It can detect partitions from the folder layout such as year=2026/month=01/.

Analogy: a librarian walking the shelves, reading the spine of each book and writing new cards.

Crawlers are convenient, but they guess. They can pick the wrong type for a column with mixed values. For stable pipelines, many teams define tables explicitly and use crawlers only for exploration or for adding partitions.

Step 4: partitions are registered

For a partitioned table, the catalog stores one entry per partition with its location. When a query filters on the partition key, the engine asks the catalog which partitions match and reads only those folders. This is partition pruning.

If you add new folders on S3 and do not tell the catalog, queries will not see them. You can register partitions with a command, a crawler run, an API call, or a feature such as partition projection that computes them from a pattern.

Step 5: engines read the catalog

When you run SELECT * FROM sales.orders, the engine asks the catalog for the table definition. It learns the schema, location and format, then reads the files directly from storage. The catalog is only consulted for metadata, not for the data bytes.

Step 6: schema changes

When columns are added or types change, the table definition must change too. With plain file tables, readers match columns by name or position depending on the format and settings. With table formats, the metadata tracks schema evolution more safely.

Step 7: permissions

Access to catalog objects is controlled by identity policies. For finer control, Lake Formation can grant permissions on databases, tables and columns, and engines that integrate with it enforce those grants at read time.

Step 8: sharing across accounts

Catalog resources can be shared with other accounts, typically through Lake Formation grants or resource links. This lets a central data team publish tables to many consumer accounts without copying data.

Key concepts and terms

A worked example

A team has order files in S3 and wants to query them by name.

Define the database and table with SQL

CREATE DATABASE IF NOT EXISTS sales_curated;

CREATE EXTERNAL TABLE sales_curated.orders (
  order_id      bigint,
  customer_id   bigint,
  amount        decimal(12,2),
  order_ts      timestamp
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://example-lake-clean/orders/'
TBLPROPERTIES ('parquet.compression'='SNAPPY');

The keyword EXTERNAL signals that the catalog owns only the definition, and the files belong to the storage location.

Register partitions

MSCK REPAIR TABLE sales_curated.orders;

This scans the location and adds partitions that exist on storage but not in the catalog. For tables with very many partitions, a more targeted approach is better.

ALTER TABLE sales_curated.orders ADD IF NOT EXISTS
  PARTITION (order_date = '2026-01-15')
  LOCATION 's3://example-lake-clean/orders/order_date=2026-01-15/';

Query by name

SELECT customer_id, SUM(amount) AS total
FROM sales_curated.orders
WHERE order_date = '2026-01-15'
GROUP BY customer_id
ORDER BY total DESC
LIMIT 10;

The filter on order_date lets the engine skip all other partitions.

Read the catalog from Python

import boto3

glue = boto3.client("glue", region_name="us-east-1")

resp = glue.get_table(DatabaseName="sales_curated", Name="orders")
table = resp["Table"]
for col in table["StorageDescriptor"]["Columns"]:
    print(col["Name"], col["Type"])
print("location:", table["StorageDescriptor"]["Location"])
print("partition keys:", [k["Name"] for k in table.get("PartitionKeys", [])])

This is useful for a pipeline that wants to build a feature list or a quality check straight from the catalog, instead of hardcoding column names.

Create a crawler

glue.create_crawler(
    Name="example-orders-crawler",
    Role="arn:aws:iam::000000000000:role/example-crawler-role",
    DatabaseName="sales_curated",
    Targets={"S3Targets": [{"Path": "s3://example-lake-clean/orders/"}]},
    SchemaChangePolicy={"UpdateBehavior": "LOG", "DeleteBehavior": "LOG"},
)

Setting update and delete behavior to log means schema changes are recorded for review instead of silently applied. That is often wise for production tables.

A short numeric walk-through of pruning

Imagine a table with 1,000 daily partitions of roughly equal size. A query with order_date = '2026-01-15' needs one partition. The engine asks the catalog for that partition's location and reads about one thousandth of the data. Without partitions, it would scan everything. With partitions but no filter, it would also scan everything. The catalog enables the saving, and the query must use the key to benefit.

A second example: catalog driven quality check

A pipeline can compare the catalog schema against a freshly written file before it publishes a partition. This catches drift early.

import boto3
import pyarrow.parquet as pq

glue = boto3.client("glue", region_name="us-east-1")
cols = glue.get_table(DatabaseName="sales_curated", Name="orders")["Table"]["StorageDescriptor"]["Columns"]
expected = {c["Name"]: c["Type"] for c in cols}

schema = pq.read_schema("part-0001.parquet")
actual = {f.name: str(f.type) for f in schema}

missing = sorted(set(expected) - set(actual))
extra = sorted(set(actual) - set(expected))
if missing or extra:
    raise SystemExit(f"schema drift: missing={missing} extra={extra}")
print("schema matches the catalog")

The check compares names only. A stricter version would also map catalog types to file types, such as bigint to int64, and fail on a mismatch. The cost of this small gate is tiny, and it prevents broken partitions from reaching dashboards or model training.

Table versions and rollback

When a table definition changes, the catalog can keep the earlier version. This lets you inspect what a table looked like when a past model run read it. It is not a substitute for data versioning, since it records the definition and not the files. For true time travel over data, use a table format with snapshots, such as Apache Iceberg.

Partition indexes and large tables

A table with hundreds of thousands of partitions makes the metadata lookup itself expensive. Two helpful tools exist.

• Partition indexes: the catalog keeps an index over chosen partition keys so filters are answered without listing all partitions.
• Partition projection: the engine computes partition values from a pattern. For example, a date range with a known path template needs no stored partition records at all.

Projection is fast and cheap to maintain, but it requires a predictable layout. Indexes work with any layout, yet they still need partitions to be registered.

Cataloging non-table assets

The catalog is table centered, but a pipeline can store small amounts of extra metadata in table parameters. Examples include the owner, the refresh schedule, the sensitivity label and the data version. Keep these keys consistent across the organization, for instance owner, sensitivity, source_system and snapshot_id. A consistent set of keys makes automated governance possible, because rules can read the same property from every table.

How a typical lake wires the pieces together

1. Files land in a raw prefix on object storage.
2. A job cleans them and writes columnar files to a curated prefix.
3. The job registers or updates the table and partition in the catalog.
4. Query engines resolve the table name through the catalog.
5. A permission layer applies grants to the catalog objects.
6. A lineage tool records the link between input and output tables.

Each step is simple on its own. The catalog is the shared reference that makes the steps agree with one another.

Strengths and limitations

Strengths

• Fully managed, so no metastore database to run.
• Shared by many AWS analytics engines, giving one definition of each table.
• Hive metastore compatible, so Spark and Hive based tools work with it.
• Crawlers make quick exploration of new data easy.
• Integrates with Lake Formation for fine grained permissions.
• Supports cross account sharing patterns.
• Keeps table versions, so earlier definitions can be inspected.
• Works with open table formats through catalog integration.

Limitations

• It only stores metadata. It does not validate that files on storage match the definition.
• Crawlers infer types by sampling and can guess wrongly on messy data.
• Tables with huge partition counts can make planning slow without projection or a table format.
• It is tied to the AWS ecosystem, though the Hive API eases tools outside it.
• It is not a business glossary. Rich descriptions, owners and ratings need conventions or a separate data catalog product.
• Lineage is not captured automatically at column level. See data lineage for tools that do this.
• Schema drift between files and definition can cause query errors that are hard to trace.

How it compares with alternatives

The Glue Data Catalog is a technical catalog. It pairs well with a business-facing catalog when the organization needs glossaries and stewardship. It also pairs with Apache Iceberg, where it can serve as the catalog that points to current table metadata.

Role in tabular foundation models

A model that learns from raw tables must first know what tables it has. The catalog supplies this knowledge. See /model/architecture for how a tabular model reads columns and rows, and /model/benchmarks for how such models are compared.

Discovery of training and scoring tables

A pre-training or evaluation pipeline can list databases and tables from the catalog, filter by properties such as domain or row count, and build a manifest. Instead of walking a bucket, it asks the catalog. This makes sampling across many tables systematic. A tag like domain=retail on a table lets the pipeline balance the mix.

Row and column structure, mixed types

The catalog holds column names and types. A tabular model needs to know which columns are numeric, categorical, date or text, because it encodes each differently. Reading types from the catalog saves a guessing step. When types in the catalog and in the files disagree, trust the files but fix the catalog, since silent mismatch can break encoding.

No column names, or meaningless ones

Many real tables have columns named c1, c2 or cryptic codes. A catalog entry can carry a comment or a property that describes a column. A tabular foundation model that does not rely on names can ignore this, while a model that can use semantics may take the comment as extra context. Either way, keeping a description in the catalog is cheap and helps both people and models.

Missing values

Catalog column statistics and profiling jobs can store null counts. A pipeline can read those to decide which columns need imputation, which tables are too sparse, and which can go straight to a model. For imputation, the catalog is a good place to record which columns were filled and by what method, as table or column parameters.

In-context learning

At inference time, a model with in-context learning needs a labeled context set from the same table or a similar one. The catalog helps a service resolve a table name to a location, schema and partitions, and then pull a small recent slice as context. Because partitions are listed in the catalog, choosing "the latest week of rows" is a metadata lookup, not a storage scan.

Continual learning

When a model keeps learning from new tables, the catalog can track which tables and partitions were already used. A property such as last_trained_partition or a separate control table provides a simple bookmark. With table formats, a snapshot identifier does the same job more precisely.

Anomaly detection

For anomaly detection, scoring jobs read partitions via the catalog and write scores into a new table also registered in the catalog. Consumers can then join the original and score tables by key. Registering the output means dashboards and alerts find it by name.

Sector and semantic classification

Predicted sector labels or semantic column types can be written back to the catalog as table or column parameters. Downstream governance can then act on them. For example, a column predicted as personal data can be tagged, and Lake Formation permissions or masking rules can be applied to the tag. This closes a loop between a model that understands tables and a system that protects them.

Governance and lineage

Because the catalog is the common entry point, it is where access control and audit meet model data. Pair it with AWS Lake Formation for permissions and with data lineage records so that every training run can say which tables, versions and columns it used.

Practical guidance and best practices

Naming and structure

• Use consistent database names by zone and domain, such as raw_, clean_, curated_ prefixes.
• Use lowercase table and column names without spaces.
• Keep one table per dataset and one schema per table.
• Add comments to tables and columns. Short descriptions pay off.

Schemas and types

• Prefer explicit table definitions for production data.
• Use crawlers for discovery and for adding partitions, with change behavior set to log or tightly controlled.
• Keep types strict. Avoid storing numbers as strings.
• Plan for schema evolution: add columns at the end, avoid renaming in place, and version breaking changes as new tables.

Partitions

• Partition by a column that most queries filter on, usually a date.
• Avoid high cardinality partition keys.
• Use partition projection when partitions follow a predictable pattern and counts are very large.
• Use partition indexes for tables with very many partitions, so lookups stay fast.
• Keep partition layout in step with storage: register, repair or project.

Security

• Grant access through roles, not users.
• Use Lake Formation for column and row level permissions and central grants.
• Separate producer and consumer accounts, and share through grants or resource links.
• Log catalog changes so you can audit who altered a table definition.

Operations

• Manage catalog objects as code with a template or infrastructure tool, so changes are reviewed.
• Run schema checks in the pipeline before registering new data.
• Monitor crawler runs and failures.
• Clean up orphan tables and test databases.

For model pipelines

• Tag tables with domain, sensitivity and freshness.
• Store a data version or snapshot in a table property for each model run.
• Record which columns were imputed and how.
• Keep a small curated database for evaluation tables so benchmarking stays repeatable.

Troubleshooting guide

Migration checklist from a self-run metastore

1. Export the existing table and partition definitions.
2. Create matching databases and tables in the managed catalog.
3. Verify that locations, formats and SerDe settings carry over.
4. Point test clusters at the new catalog and run a set of known queries.
5. Compare row counts and sample values against the old setup.
6. Move permissions to roles and, where needed, to Lake Formation grants.
7. Switch production engines in stages, keeping the old metastore for a rollback window.

Documenting tables for people and models

A short description on each table should answer four questions: what does a row represent, what is the grain (one row per what), how fresh is it, and who owns it. A short description on each column should state the unit, the allowed values and how missing values are written. These few sentences make a table usable by a new analyst and also supply useful context for any model that can read text about a column.

Environments and promotion

Keep development, test and production catalogs separate, either as different accounts or as clearly named databases. Promote table definitions the same way you promote code.

• Define tables in version controlled files.
• Review changes in a pull request style process.
• Apply to test first and run sample queries.
• Apply to production with the same files.
• Record the version of the definitions in a table property.

This habit prevents the common situation where the production table was edited by hand and nobody remembers why.

Naming for model pipelines

Models that read many tables benefit from predictable names. A convention such as <zone>_<domain>.<entity>_<grain> makes it easy for code to select all curated tables of a domain at daily grain. Predictable names also make evaluation sets easy to define, since a benchmark can list its tables by pattern.

Freshness signals

Add a property such as last_updated or max_partition that the loading job refreshes after each successful run. Scoring services can check it before they run, and skip or warn when data is stale. Without such a signal, a model may silently score on old data.

Common mistakes

1. Assuming the catalog validates data: it does not check that files match the schema.
2. Letting crawlers run wild on production: schema changes can ripple into dashboards and models.
3. Forgetting to register partitions: new folders exist on storage but queries return no rows.
4. Over-partitioning: too many tiny partitions slow planning and reading.
5. Dropping a table and expecting data to vanish: for external tables the files remain.
6. Mixed formats under one location: a crawler or engine may fail or pick the wrong format.
7. Weak types: storing everything as string pushes errors downstream.
8. No ownership info: nobody knows who to ask about a table.
9. Hand edits in the console only: changes are not reviewed or repeatable.
10. Skipping descriptions: models and people both lose useful context.
11. Ignoring cross account permissions: sharing fails because the storage and key permissions do not match the catalog grant.
12. Using the catalog as the only governance tool: add permissions, tagging and lineage on top.

Frequently asked questions

What is the AWS Glue Data Catalog used for?

It is used to store table definitions, schemas, locations and partitions so that query and processing engines can find data by name. It acts as a central, managed metadata store shared by services such as Athena and EMR.

Does the Glue Data Catalog store my data?

No. It stores only metadata. Your data stays in storage such as S3, and the catalog points to it. Deleting a table definition normally leaves the underlying files in place.

Do I need a crawler to use the catalog?

No. You can create databases and tables yourself with SQL, an API or infrastructure code. Crawlers are an optional convenience that infer schemas and register partitions by scanning data.

Is the catalog compatible with Hive and Spark?

Yes. It implements the Hive metastore interface, so Spark, Hive and similar tools can use it as their metastore. That makes migration from a self-run metastore much easier.

How are partitions handled?

Each partition of a table has a record with its own location. Engines ask the catalog for matching partitions when a query filters on the partition key. You must register new partitions through a crawler, a command, an API call, or avoid the step by using partition projection.

Can the catalog work with Iceberg and other table formats?

Yes. It can hold the pointer to the current metadata of an Iceberg table, so engines resolve the table by name and then read its snapshots. Other table formats can be registered in similar ways depending on the engine.

How do I control who can see which columns?

Use identity policies for basic access and Lake Formation for fine grained rules at database, table, column and row level. Engines that integrate with Lake Formation enforce those rules when they read.

Is the Glue Data Catalog a business glossary?

Not really. It is a technical catalog focused on schemas and locations. You can add comments and parameters, but a glossary with owners and definitions is often managed in a separate data catalog tool or a documented convention.

How does it help machine learning on tables?

It gives pipelines a reliable list of tables, column names, types and partitions, so they can build datasets, check schemas and record which data versions a model used. It can also store outputs such as imputation notes or predicted semantic labels as properties.

What happens if the schema in the catalog differs from the files?

Queries may fail, return nulls or return wrong values, depending on the format and engine settings. The fix is to update the catalog or the files so they agree, and to add checks in the pipeline that compare them before data is published.

Glossary

• Metadata: data that describes other data.
• Catalog: an index of datasets and their descriptions.
• Metastore: the store that keeps table definitions for query engines.
• External table: a table whose data is owned by the storage location, not the catalog.
• Crawler: a tool that scans data and infers schema.
• Partition: a subset of a table stored under its own path.
• Partition pruning: skipping partitions that cannot match the filter.
• Partition projection: computing partition values at query time from a rule.
• SerDe: the library that reads and writes a given file format.
• DDL: data definition language, the SQL used to create tables.
• Schema evolution: allowed changes to columns over time.
• Resource link: a pointer to a shared catalog object.
• Column statistics: summary numbers about column values.
• Lineage: the record of where data came from and how it changed.
• Governance: policies and controls for data access and quality.
• Tabular foundation model: a model pre-trained on many tables to work on new ones.
• Imputation: filling in missing values.
• Semantic type: the meaning of a column, such as a date, an address or an identifier.

Summary

The AWS Glue Data Catalog is a managed, Hive compatible metadata repository. It records databases, tables, columns, partitions and locations, and it lets many engines share one definition of each table. It does not store or check the data. Good results come from explicit schemas, careful partition handling, controlled crawlers and permissions applied through Lake Formation.

For tabular foundation models, the catalog is the discovery and bookkeeping layer. It lists the tables that can feed training or scoring, supplies column names and types, holds null statistics that guide imputation, resolves partitions for in-context slices, and records which snapshots or partitions a continual learner has seen. Model outputs such as anomaly scores or semantic labels can be registered back into it, so people and systems find them by name. See also Amazon S3, AWS Lake Formation, Amazon Athena and /model/research for the surrounding pieces.

Related

It belongs to AWS data analytics in my stack, under Data engineering and lakehouse. Others in the same group: