AWS Lake Formation

Overview

AWS Lake Formation is a service that helps you build and secure a data lake on AWS. Its most important feature is central access control. You grant permissions on databases, tables, columns and rows in one place, and integrated engines enforce those grants when users query the data. It works on top of the AWS Glue Data Catalog and the files in Amazon S3.

Without such a layer, you must manage access through many separate storage policies and identity policies, which becomes hard as the lake grows. Lake Formation gives a database style permission model, with grants and revokes, to a lake. For teams that feed tables to models, including the tabular foundation models described on /model/research, it is the control point that decides which columns and rows a pipeline or a person may see.

What it is

Lake Formation is a governance and setup service. It has three broad jobs.

1. Set up a lake: register storage locations, define databases and tables, and ingest data with the help of blueprints and jobs.
2. Secure a lake: manage permissions centrally, at the level of database, table, column, row and cell.
3. Share a lake: let other accounts or organizations use selected tables without copying data.

An everyday picture helps. Imagine a large office building with many rooms of documents. Without a central system, each room has its own key and its own rules. Lake Formation is the front desk with a badge system. You tell the desk who may enter which room, which shelves and which pages, and the desk applies the rules everywhere.

Main building blocks

• Data lake administrator: a principal allowed to manage the lake and grant permissions.
• Registered location: an S3 path that Lake Formation manages access for.
• Principal: a user, role, group or account that receives permissions.
• Permission: a right such as SELECT, INSERT, ALTER, DROP or DESCRIBE on a resource.
• Data filter: a rule that restricts columns and rows for a grant.
• LF-tag: a key and value label attached to resources and used to grant by attribute.
• Credential vending: the way integrated engines receive temporary access to the data for an authorized query.

How it differs from plain IAM and bucket policies

Identity and bucket policies can control who reads which S3 path. That works for coarse rules. It is a poor fit for rules such as "analysts can see everything in the orders table except the email column, and only rows for their own region". Lake Formation expresses these rules at the level of data, not at the level of file paths.

Where it came from

Lakes began as buckets of files. Early governance meant folder level policies and manual copies of data with sensitive fields removed. Copies drifted out of date and multiplied. Teams wanted the permission style of a database: grant select on a table, hide a column, filter rows.

Lake Formation brought that style to lakes on AWS. It first focused on simplifying setup and on central permissions. Later it added more flexible ways to scale governance:

• Tag based access control, so a handful of tags can cover thousands of tables.
• Row and cell level filters.
• Cross account sharing with the Glue catalog as the shared layer.
• Support for governed and then open table formats, as the industry moved toward Apache Iceberg and Delta Lake.

The broader trend was the shift from "secure the storage" to "secure the data", with the catalog as the anchor.

How it works

Step 1: register a location

You register an S3 path with Lake Formation using a role that can read and write there. From then on, Lake Formation can control access for that path. Data in registered locations is meant to be accessed through the catalog, not by direct file access, for users who are governed.

Analogy: handing the building's master key to the front desk, and agreeing that visitors go through the desk.

Step 2: define catalog resources

Databases and tables live in the Glue Data Catalog. Lake Formation grants apply to those catalog objects. A table points to a registered location.

Step 3: grant permissions

An administrator grants a principal a permission on a resource. Examples:

• SELECT on sales_curated.orders for an analyst role.
• DESCRIBE on a database for a data science role.
• ALTER and INSERT on a staging table for an ingestion role.

Grants can include a grant option, which lets the receiver pass on the same right.

Step 4: narrow with columns, rows and cells

A SELECT grant can include or exclude certain columns. A data filter can also add a row condition. The combination is a cell level rule: the user sees only certain columns for certain rows.

Analogy: a visitor is allowed into the records room, may read only the "name" and "amount" pages, and only for customers in their own city.

Step 5: use tags for scale

With thousands of tables, granting one by one is slow. LF-tags solve this.

1. Define a tag key such as sensitivity with values like public, internal, restricted.
2. Attach tags to databases, tables or columns.
3. Grant a principal access to resources that match a tag expression, such as sensitivity = public.

When a new table is tagged, matching principals gain access automatically. When a column is tagged restricted, those without that tag expression lose it. This is attribute based access control for data.

Step 6: query through an integrated engine

When a user runs a query in an integrated engine, the engine checks with Lake Formation. If the user is allowed, the engine gets temporary, scoped access to the underlying files and returns only permitted columns and rows. If not, the query fails or returns a filtered view.

Several AWS engines integrate this way, including Amazon Athena and Amazon EMR. Some integrations differ in which features they support, so check the compatibility of your engine and version before relying on a rule.

Step 7: share across accounts

An owner account can grant permissions to another account or organization. The consumer sees shared databases or tables through resource links in their own catalog and can then grant access to their own principals. Data is not copied.

Step 8: audit

Access and permission changes are logged through the audit service of the cloud account. Reviewing these logs shows who queried what and who granted what.

Key concepts and terms

A worked example

A company has an orders table with customer emails and a region column. Analysts in each region should see order amounts and regions but not email addresses, and only rows for their own region.

Register the location

import boto3

lf = boto3.client("lakeformation", region_name="us-east-1")

lf.register_resource(
    ResourceArn="arn:aws:s3:::example-lake-clean/orders",
    UseServiceLinkedRole=True,
)

Create a data filter

lf.create_data_cells_filter(
    TableData={
        "TableCatalogId": "000000000000",
        "DatabaseName": "sales_curated",
        "TableName": "orders",
        "Name": "orders_emea_no_email",
        "RowFilter": {"FilterExpression": "region = 'EMEA'"},
        "ColumnWildcard": {"ExcludedColumnNames": ["email"]},
    }
)

This filter hides the email column and keeps only rows where the region is EMEA.

Grant the filter to a role

lf.grant_permissions(
    Principal={"DataLakePrincipalIdentifier": "arn:aws:iam::000000000000:role/example-analyst-emea"},
    Resource={
        "DataCellsFilter": {
            "TableCatalogId": "000000000000",
            "DatabaseName": "sales_curated",
            "TableName": "orders",
            "Name": "orders_emea_no_email",
        }
    },
    Permissions=["SELECT"],
)

What the analyst sees

SELECT region, COUNT(*) AS n, SUM(amount) AS total
FROM sales_curated.orders
GROUP BY region;

The result contains only the EMEA row group. A query that selects email fails or returns nothing for that column, depending on the engine.

Scale it with tags

Instead of one filter per region and table, use tags for the column rule.

lf.create_lf_tag(TagKey="sensitivity", TagValues=["public", "internal", "restricted"])

lf.add_lf_tags_to_resource(
    Resource={"TableWithColumns": {
        "DatabaseName": "sales_curated",
        "Name": "orders",
        "ColumnNames": ["email"],
    }},
    LFTags=[{"TagKey": "sensitivity", "TagValues": ["restricted"]}],
)

lf.grant_permissions(
    Principal={"DataLakePrincipalIdentifier": "arn:aws:iam::000000000000:role/example-analyst"},
    Resource={"LFTagPolicy": {
        "ResourceType": "COLUMN",
        "Expression": [{"TagKey": "sensitivity", "TagValues": ["public", "internal"]}],
    }},
    Permissions=["SELECT"],
)

Now any column tagged restricted, in any table, is hidden from this role, and any new column tagged public is visible without new grants.

A tiny numeric picture

Suppose the lake has 2,000 tables with an average of 25 columns each, and 10 analyst groups. Granting by name would need up to 500,000 column level grants, which no team can manage. With tags, you maintain a few tag values and a handful of expressions per group, and the labels on columns do the rest. The work moves from the number of grants to the quality of the labels.

A second example: designing a tag taxonomy

A tag taxonomy is the heart of a scalable setup. A small, clear design looks like this.

Roles then map to tag expressions.

# Illustrative policy table, not a real configuration format
roles:
  analyst-sales:
    select_when: "domain = sales AND sensitivity IN (public, internal)"
  data-scientist:
    select_when: "training_allowed = yes AND pii = none"
  steward:
    select_when: "sensitivity IN (public, internal, restricted)"
    grant_option: true

Each rule fits on one line, and a reviewer can read it in seconds. When a new table arrives, it needs only four tags to be correctly covered.

Step by step: onboarding a new dataset

1. A producer lands files in a registered location under a raw zone.
2. The producer defines the table in the catalog.
3. A steward tags the database or table and marks sensitive columns.
4. A pipeline job cleans the data and writes a curated table that inherits the tags.
5. Consumers receive access through tag expressions, with no new grants needed.
6. A scheduled report lists tables with missing tags, so gaps are closed quickly.

This flow makes governance a part of delivery, not a separate project after the fact.

Hybrid mode and migration

Many lakes already have permissions based on identity and storage policies. Switching everything at once is risky. Hybrid access mode lets you move table by table.

• Keep old access working while you add Lake Formation grants for pilot tables.
• Verify with test roles that column and row rules behave as planned.
• Remove the broad storage access for the pilot tables.
• Repeat for the next group of tables.

At the end of the migration, direct storage access exists only for a few service roles that need it, such as the ingestion job and the engine integration role.

Testing permissions

Treat access rules as code and test them.

def can_read(role_session, table, column):
    try:
        q = f"SELECT {column} FROM {table} LIMIT 1"
        run_query_as(role_session, q)
        return True
    except PermissionError:
        return False

assert can_read(analyst, "sales_curated.orders", "amount") is True
assert can_read(analyst, "sales_curated.orders", "email") is False

Here run_query_as stands for any helper in your environment that executes a query using the given role. The point is the habit: for each role, list the columns that must be visible and the columns that must be hidden, and run those checks after every change to tags or filters.

Typical failure modes seen in practice

Governance for tables that models write

Pipelines that write predictions create new data assets. Define from the start who may read them. A good default is to copy the strictest tag from any input column, then let a steward relax it when appropriate. This prevents a sensitive source from leaking through a score that looks harmless, such as a risk score built from health or income data.

Strengths and limitations

Strengths

• One place for permissions across many engines and accounts.
• Fine grained control at column, row and cell level.
• Tag based access scales to large lakes.
• Familiar grant and revoke model, close to a relational database.
• Cross account sharing without copying data.
• Works with the shared Glue catalog, so table definitions and permissions stay together.
• Central audit trail.
• Reduces the need for many masked copies of the same data.

Limitations

• Only engines that integrate with it enforce the rules. A tool that reads files directly with a broad storage role bypasses it.
• Feature support differs by engine, table format and version, so test each combination.
• Mixing old style storage policies and Lake Formation grants can confuse debugging, since both layers must allow access.
• Tag design takes planning. Poor tag taxonomy causes either over-sharing or constant exceptions.
• Row filters add query overhead and add complexity in reasoning about what a user sees.
• It governs access, not data quality. It does not detect bad values or bias.
• Learning curve: principals, tags, filters, registration and hybrid modes all interact.

How it compares with alternatives

The most robust setups combine layers. Lake Formation handles data level rules. Identity policies limit who can assume roles. Bucket policies block direct access outside approved paths. Encryption keys add one more barrier.

Compare also with metadata-only tools such as a business data catalog and with data lineage systems. Those help people find and trace data. Lake Formation decides who may read it.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns, handle mixed types and missing values, and may see no useful column names. See /model/architecture for how they represent a table. The data they touch is often sensitive, which is why governance matters. Lake Formation relates to such models in these concrete ways.

Deciding which tables may enter training

Pre-training and fine tuning need many tables. Some tables must never be used, such as those with personal data without consent. A tag like training_allowed = yes lets the pipeline's role read only approved tables. A new table is excluded by default until someone tags it. This is far safer than listing paths in a config file.

Column level protection of sensitive fields

Tabular models do not need names, emails or exact identifiers to learn patterns. Hide those columns with column level grants or restricted tags for the training role. The model sees the numeric and categorical structure, while the sensitive text never leaves its boundary. If a column must be kept in some form, a hashed or bucketed version can be created upstream and tagged as internal.

Row level filters for context sets

For in-context learning, the model reads a labeled set of rows as context. A row filter ensures that context comes only from rows the caller may see. For instance, a service acting for one business unit gets context rows filtered to that unit. This prevents leakage across tenants through the context window.

Missing values and imputation outputs

When a pipeline imputes values, it writes a new column or table. Lake Formation permissions and tags should apply to the output as well, and an imputed column should inherit the sensitivity of its source. A simple rule is to copy the tags from the source column to the output column in the same job.

Anomaly detection and sensitive findings

Scores from anomaly detection can reveal facts about individuals, such as unusual spending. Store them in their own table with the same or tighter permissions as the source, and expose summaries to wider audiences if needed.

Sector and semantic classification

A model that predicts the sector of a table or the semantic type of a column can write its result as an LF-tag. A column predicted as an email, a phone number or a national identifier can be tagged restricted automatically, and the access rules then apply without manual review of every table. Treat such automatic tags as suggestions that a steward confirms, since models can be wrong, and a wrong tag in the unsafe direction leaks data.

Continual learning and audit

For models that keep learning from new tables, the audit log shows which roles read which tables and when. Combined with snapshot identifiers from Apache Iceberg and with lineage records, this lets you answer the question "which data did this model version see?" That is vital for deletion requests and for explaining a model's data sources.

Sharing evaluation data

Benchmarking, as covered on /model/benchmarks, often needs a shared set of tables. Cross account sharing lets an evaluation account read a fixed, governed set of tables without copies. Because the data stays in one place, every evaluator sees the same version.

Practical guidance and best practices

Plan the model first

• List data domains, sensitivity levels and consumer groups.
• Define a small, stable tag taxonomy. Three to five keys are usually enough.
• Decide who owns tags, who may change them and how changes are reviewed.

Set up carefully

• Register locations with a dedicated role.
• Separate raw, curated and shared zones.
• Block direct storage access for regular users, so the lake permission layer cannot be bypassed.
• Use hybrid mode when migrating, then tighten.

Use tags rather than named grants where possible

• Tag at the database level for broad defaults, and override at the column level for sensitive fields.
• Grant to roles that represent job functions, not to individuals.
• Review tag assignments regularly. Untagged resources are a risk, so add a rule that new tables must be tagged before use.

Row and cell filters

• Keep filter expressions simple and tested.
• Test as the target role, not as an administrator.
• Check performance on representative queries.
• Document what each filter means in business terms.

Cross account sharing

• Share through grants and resource links instead of copying.
• Make sure storage and encryption key permissions allow the consumer path.
• Keep a list of consumers and review it.

Audit and monitoring

• Turn on logging for data access and permission changes.
• Alert on unusual grants, such as broad SELECT on restricted tags.
• Keep evidence of reviews for compliance.

For model pipelines

• Give the training role the minimum tag set.
• Run training jobs in engines that enforce the rules.
• Propagate tags to derived tables.
• Store the grant set and data snapshot with the model record.
• Test that the model pipeline fails closed: if tags are missing, nothing is readable.

Common mistakes

1. Leaving a back door: broad storage permissions let users read files directly and ignore column rules.
2. Granting to individuals: it does not scale and makes audits painful.
3. Poor tag taxonomy: too many keys or unclear values cause drift and exceptions.
4. Forgetting derived tables: a copy of a sensitive column in a new table loses protection.
5. Testing as admin: rules look fine for the administrator but are wrong for real users.
6. Assuming every engine enforces rules: unsupported engines may deny access or bypass checks, depending on setup.
7. Not controlling the encryption key: if the key policy is open, the layered defense is weaker.
8. Using governance as a quality tool: permissions do not fix wrong data.
9. Ignoring row filter cost: complex filters can slow queries.
10. No review cycle: permissions accumulate and are never removed.
11. Trusting automatic labels blindly: a model driven tag can be wrong. Add review for sensitive classes.
12. Skipping documentation: nobody knows why a filter exists, so nobody dares to change it.

Frequently asked questions

What does AWS Lake Formation do?

It lets you set up a data lake and manage access to it from one place. You grant permissions on databases, tables, columns and rows, and integrated engines enforce them when users query the data.

How is Lake Formation different from IAM policies?

IAM policies control which actions an identity may call, often at the level of services and storage paths. Lake Formation controls access at the level of data objects, including columns and rows. Both layers apply, and a request needs to be allowed by each relevant layer.

What are LF-tags?

LF-tags are key and value labels you attach to databases, tables and columns. You grant access by tag expression, such as everything labeled public. New resources with the right label are covered automatically, which is why tags scale far better than named grants.

Can I hide a single column from some users?

Yes. A grant can include only certain columns, or exclude certain ones, and a tag can mark a column as restricted so that principals without matching access do not see it. The enforcement happens in engines that integrate with Lake Formation.

Can Lake Formation filter rows?

Yes. A data filter holds a row condition, and optionally a column list. Granting SELECT through that filter gives a user only the matching rows, and with columns excluded it becomes cell level security.

Does Lake Formation work without the Glue Data Catalog?

It is built around the Glue Data Catalog. Permissions attach to databases and tables in that catalog, so you normally define your tables there first.

Is data copied when I share it with another account?

No. Sharing creates grants and resource links, and the consumer reads the same underlying files under the granted permissions. This keeps one copy of the data and one place to revoke access.

What happens if a user reads files directly from S3?

If their storage permissions allow it, they can bypass the table level rules. That is why you should restrict direct storage access for regular users and route them through governed engines.

How does it help with machine learning on sensitive tables?

It lets you give a training or scoring role access to only approved tables, hide sensitive columns, and filter rows per tenant. Together with audit logs, it also helps you show which data a model could read.

Does Lake Formation check data quality?

No. It controls who can access data, not whether the data is correct. Quality checks, profiling and validation need separate tools and pipeline steps.

Glossary

• Governance: the policies and controls that decide how data may be used.
• Principal: an identity that can receive permissions.
• Grant: giving a permission to a principal.
• Revoke: removing a permission.
• Data filter: a rule that limits columns and rows for a grant.
• Cell level security: restricting both which rows and which columns a user can see.
• LF-tag: a label used to grant access by attribute.
• Attribute based access control: granting rights based on labels rather than named resources.
• Registered location: a storage path managed by the lake permission layer.
• Credential vending: temporary storage access issued to approved engines.
• Resource link: a pointer to a shared catalog object.
• Hybrid access mode: a transition mode that supports old and new access methods.
• Least privilege: giving only the access needed for a task.
• Audit trail: a log of access and changes.
• PII: personally identifiable information.
• Derived table: a table created from other tables, which should inherit sensitivity.
• Fail closed: deny access when information is missing.
• Tabular foundation model: a model pre-trained on many tables to work on new ones.

Summary

AWS Lake Formation brings a database style permission model to data lakes on AWS. It registers storage, builds on the Glue Data Catalog, and lets administrators grant access to databases, tables, columns, rows and cells. Tags make governance scale, cross account sharing avoids copies, and integrated engines enforce the rules while logs record what happened. Its limits are clear: only integrated engines enforce it, tag design needs care, and it does not judge data quality.

For tabular foundation models, Lake Formation is the gate in front of the tables that models learn from and score. It decides which tables may enter training, hides sensitive columns, limits context rows in in-context learning, carries sensitivity tags onto imputed and derived outputs, and provides audit trails for continual learning. A semantic classifier can propose tags, with human review for sensitive classes. Learn more on the related pages for the AWS Glue Data Catalog, Amazon S3, Amazon EMR and /model/architecture.

Related

It belongs to AWS data analytics in my stack, under Data engineering and lakehouse. Others in the same group: