Amazon S3

Overview

Amazon S3 is a managed object store from AWS. It keeps files of any size and format inside containers called buckets, and it returns each file by a name called a key. It is built for very high durability, near unlimited capacity and simple access over HTTP. Because of this, it became the default storage layer for data lakes, backups, logs, model checkpoints and raw data landing zones.

For tabular work, S3 is rarely the place where tables are "queried". It is the place where table files live, usually as Parquet, ORC, CSV or JSON. Engines such as Apache Spark, Trino and Amazon Athena read those files, and catalogs such as the AWS Glue Data Catalog give them names and schemas. Models that learn from raw tables, such as those described on /model/research, usually receive their training and inference data from S3 in one form or another.

What it is

S3 stands for Simple Storage Service. It is an object store, which is different from a file system and from a block device.

• A file system has folders, permissions per file, and in-place edits.
• A block device is a raw disk attached to one machine.
• An object store holds whole objects. You write an object, read an object, replace it or delete it. You do not edit a few bytes in the middle.

The main building blocks are simple.

• Bucket: a top level container with a globally unique name inside a partition of the cloud.
• Object: the data plus metadata. The data is a sequence of bytes.
• Key: the full name of the object inside the bucket, such as raw/orders/2026/01/part-0001.parquet.
• Prefix: the leading part of a key. It looks like a folder but is only a string.
• Metadata: system fields (size, content type, last modified) and user fields.
• Version: when versioning is on, every overwrite creates a new version of the same key.

An everyday analogy helps. Think of a huge warehouse with sealed boxes. Each box has a label (the key). You can ask the clerk for a box by label, hand in a new box, or throw one away. You cannot open a box and change a single page inside. If you want a different page, you hand in a whole new box with the same label.

Why it became the center of the lake

Three properties explain the popularity of S3 for data work.

1. Durability and availability: data is stored redundantly across multiple facilities in a region, and the service is designed for extremely high durability.
2. Separation of storage and compute: you can store petabytes without running a single server, and spin up compute only when you need to analyze.
3. Open formats: because S3 stores plain files, many tools can read the same data without moving it.

Where it came from

S3 was among the first public AWS services and launched in the mid 2000s. Its early role was simple storage for web assets and backups. As big data tools matured, teams noticed that cheap, durable storage combined with columnar files was enough to build analytic systems.

A short timeline of ideas, without exact dates:

• Early Hadoop clusters used HDFS, a distributed file system tied to compute nodes. Storage and compute scaled together.
• Cloud object stores offered an alternative where storage scaled separately. Connectors let Hadoop era tools read from S3.
• Columnar formats such as Parquet and ORC made scans over object storage efficient by reading only the needed columns.
• Table formats such as Apache Iceberg and Delta Lake added transactions and schema evolution on top of plain objects.
• S3 moved from eventual consistency for some operations to strong read after write consistency, which removed many workarounds in data tools.

Over the years the service added storage classes for cold data, event notifications, lifecycle rules, replication, access points and tools for inventory and analytics. The core idea of buckets, keys and objects stayed the same.

How it works

This section walks through what happens when data moves in and out of S3, in plain steps.

Step 1: you create a bucket

A bucket is created in one region. Its name must be unique. You choose settings such as default encryption, versioning and public access blocking. Good defaults block public access and turn on encryption at rest.

Analogy: you rent a section of the warehouse and put a lock on its door.

Step 2: you write an object

You call an API with a bucket, a key and a body. For small objects, one request is enough. For large objects, you use multipart upload.

• The file is split into parts.
• Parts upload in parallel, and failed parts can be retried alone.
• A final call stitches the parts into one object.

Analogy: moving a large piece of furniture in several trips and having the clerk reassemble it at the end.

Step 3: S3 stores it redundantly

The service stores copies across several devices and facilities in the region. You do not see this. You only see that the write succeeded or failed.

Step 4: you read it

A read is a GET request by key. Reads can request a byte range, which is important for tabular data. A Parquet reader fetches the footer first to learn where each column chunk lives, then requests only the byte ranges it needs.

Analogy: asking the clerk for page 40 to 60 of a thick binder rather than the whole binder.

Step 5: you list it

A LIST request returns keys that share a prefix, in pages. Listing is slower than reading and is a classic source of trouble in data jobs that have millions of small files. Table formats avoid heavy listing by tracking files in manifests.

Step 6: consistency

S3 gives strong read after write consistency for new objects, overwrites and deletes, and for list operations. A job that writes a file and then lists the prefix will see it. This property made committers for big data engines simpler, although atomic rename is still not a native operation. Renaming in S3 is a copy followed by a delete.

Step 7: security checks

Every request is checked against several layers.

• Identity policies attached to the caller.
• Bucket policies attached to the bucket.
• Access point policies, if used.
• Block public access settings.
• Optional object ownership and ACL settings.
• Encryption key permissions, if a customer managed key is used.

A request is allowed only when the combination permits it and nothing denies it.

Step 8: lifecycle and replication

Rules can move objects to colder classes, expire old versions, abort incomplete multipart uploads and replicate objects to another bucket or region. These run in the background.

Storage classes in short

Pick the class by how often data is read and how fast you need it back. Do not choose by guesswork, since retrieval and minimum duration rules differ between classes.

Key concepts and terms

A worked example

Suppose a team receives daily order extracts and wants a clean, queryable lake. The steps below show a small but realistic flow.

Layout

Use a clear key scheme. Partition by date so engines can skip data.

s3://example-lake-raw/orders/ingest_date=2026-01-15/part-0001.csv.gz
s3://example-lake-clean/orders/order_date=2026-01-15/part-0001.parquet
s3://example-lake-clean/orders/order_date=2026-01-15/part-0002.parquet

Upload with Python

import boto3

s3 = boto3.client("s3", region_name="us-east-1")

s3.upload_file(
    Filename="orders_2026-01-15.csv.gz",
    Bucket="example-lake-raw",
    Key="orders/ingest_date=2026-01-15/part-0001.csv.gz",
    ExtraArgs={"ServerSideEncryption": "aws:kms"},
)

The upload_file helper switches to multipart upload for large files automatically.

Convert to Parquet

import pandas as pd

df = pd.read_csv("s3://example-lake-raw/orders/ingest_date=2026-01-15/part-0001.csv.gz")
df["order_date"] = pd.to_datetime(df["order_ts"]).dt.date.astype("string")
df.to_parquet(
    "s3://example-lake-clean/orders/",
    partition_cols=["order_date"],
    index=False,
)

Parquet stores each column together, compresses well and carries a schema in its footer.

Read only what you need

import pyarrow.dataset as ds

dataset = ds.dataset("s3://example-lake-clean/orders/", format="parquet", partitioning="hive")
table = dataset.to_table(
    columns=["order_id", "amount", "customer_id"],
    filter=ds.field("order_date") == "2026-01-15",
)
print(table.num_rows)

Two savings happen here. Partition pruning skips other dates. Column pruning reads only three columns from the files that remain.

Add a lifecycle rule

{
  "Rules": [
    {
      "ID": "raw-to-cold",
      "Status": "Enabled",
      "Filter": {"Prefix": "orders/"},
      "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"}],
      "Expiration": {"Days": 400},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
    }
  ]
}

This moves older raw files to a cheaper class, deletes them after a retention window, and cleans up failed uploads.

A quick size walk-through

Say a day has 10 million rows and each row is about 200 bytes in CSV. That is roughly 2 GB raw. After compression and columnar encoding, the Parquet version is often several times smaller, but the exact ratio depends on the data. A query that needs 3 of 20 columns may read well under a fifth of the Parquet bytes. This is why file format matters as much as the store itself.

A second example: event-driven validation

New files often need a check before they reach the clean zone. S3 can send an event when an object is created. A small function or queue consumer then validates the file.

# Example notification configuration (illustrative)
bucket: example-lake-raw
events:
  - type: object-created
    prefix: orders/
    suffix: .csv.gz
    target: queue-example-validate-orders

The consumer can run simple checks.

• Is the file readable and non-empty?
• Do the column names and types match the expected schema?
• Is the share of nulls in key columns within a normal range?
• Is the row count close to the previous day?

import pandas as pd

def validate(path: str) -> list[str]:
    problems = []
    df = pd.read_csv(path, nrows=100000)
    expected = {"order_id", "order_ts", "amount", "customer_id"}
    missing = expected - set(df.columns)
    if missing:
        problems.append(f"missing columns: {sorted(missing)}")
    if "amount" in df and df["amount"].isna().mean() > 0.05:
        problems.append("more than 5 percent null amounts")
    return problems

If problems appear, move the file to a quarantine prefix and notify the owner. If not, promote it to the clean zone. This keeps bad data out of model training sets, which matters for any learner that trusts its inputs.

Reading the request and cost model

S3 charges are generally built from a few parts. Knowing the parts helps you design well, without needing any specific price.

• Storage: the bytes held, by class and per month.
• Requests: PUT, COPY, POST, LIST and GET calls. Many tiny files mean many requests.
• Retrieval: for colder classes, reading data back can carry a retrieval charge.
• Data transfer: moving data out of a region or to the internet.
• Management features: inventory, analytics and similar add-ons.

A design that writes fewer, larger files lowers request counts. A design that keeps compute in the same region lowers transfer. A design that moves cold data on a schedule lowers storage.

Performance habits worth knowing

• S3 scales request rates by prefix, so spreading hot data across several prefixes helps at very high rates.
• Parallel byte range reads of a large file can raise throughput, because each request is limited but many can run at once.
• Reading many objects in parallel from many workers is the normal pattern for Spark and similar engines.
• Client side retries with backoff are essential, because transient errors and throttling responses do happen under load.
• Keep connection pools warm in long running jobs to avoid repeated handshakes.

Data integrity checks

S3 supports checksums so that you can verify an upload or download. A good pipeline records a checksum or object version next to each ingested file. When the same file is delivered twice, the pipeline can detect the duplicate rather than loading it again. For model training, this prevents duplicated rows from silently inflating one source of data.

Strengths and limitations

Strengths

• Very high durability and wide availability.
• Practically unlimited capacity, with no capacity planning.
• Storage is separate from compute, so you pay for compute only when you analyze.
• Works with almost every analytics engine and ML framework.
• Rich security controls: encryption, policies, access points, object lock.
• Lifecycle rules reduce manual cleanup.
• Event notifications allow reactive pipelines.
• Byte range reads fit columnar formats well.

Limitations

• Objects are immutable. Updating a few rows means rewriting a file.
• No native atomic rename or multi file transaction. Table formats exist to fix this.
• Listing very large prefixes is slow.
• Small files hurt read performance and raise request counts.
• Latency per request is higher than local disks, so many tiny reads are costly.
• Access control can become complex when many policy layers overlap.
• Data transfer out of a region or to the internet has a cost and a delay.
• It is not a database. It has no query language of its own for table semantics.

How it compares with alternatives

A practical view: S3 alone is the cheapest and most flexible tier. Adding a table format such as Apache Iceberg or Delta Lake gives warehouse like behavior on the same bytes. Warehouses such as Snowflake can also read open files from object storage in some setups.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns, mixed types, missing values and often no helpful column names. See /model/architecture for how such a model reads a table, and /model/research for the research background. S3 relates to these models in several concrete ways, even though it does not run any model itself.

Holding the training corpus

Pre-training on many tables needs a large library of them. Real tables are usually stored as Parquet or CSV. Synthetic tables, generated from random structural processes, can also be written to Parquet shards. S3 holds both kinds, with a key layout such as corpus/source=real/domain=retail/part-0001.parquet. The metadata in the key makes it easy to sample a balanced mix.

Preserving row and column structure

Parquet keeps the schema, types and column order. A tabular model needs to know which values are numbers, categories, dates or text. Storing tables in typed columnar files avoids losing that information, which happens quickly with plain CSV. If column names are missing or meaningless, the model can still use the types and statistics, but only if the types survive storage.

Mixed types and missing values

CSV turns everything into strings, and a missing value can be an empty field, the word NA, a dash or a zero. Parquet stores nulls explicitly. For imputation work, an explicit null is far more reliable than a guess about sentinel values. Keep a documented rule for how nulls are written when files are produced.

In-context learning and inference data

A model that uses in-context learning reads a labeled set of rows as context, then predicts for new rows. The context rows must be fetched quickly and consistently. Reading a small, well-partitioned slice from S3 by byte range is a good fit. For repeated calls, cache the context locally or in memory, since each S3 request has latency.

Continual learning and versioning

When a model keeps learning from new tables over time, you need to know exactly which data it saw. S3 versioning and immutable partition paths let you freeze a dataset snapshot. A table format snapshot ID is even better, since it names a consistent set of files. This supports reproducibility and rollback if a new batch of data hurts quality.

Imputation and anomaly detection

Both tasks run over large tables. Batch jobs in Apache Spark or Amazon EMR can read partitions from S3, call a model on chunks of rows, and write results back as new Parquet files. For anomaly detection, results such as scores and flags can be written next to the source partitions, so downstream tools can join them by key. For imputed values, write the original and the filled column side by side with a flag, so you never lose the raw value.

Sector and semantic classification

Classifying a table or a column into a sector or a semantic type uses column values, names and statistics. A catalog such as the AWS Glue Data Catalog can store the predicted labels as table or column properties. S3 provides the sample files the classifier reads. Tags on buckets or prefixes can also carry labels for governance, such as marking a prefix as containing personal data.

Governance around model data

Model data often includes sensitive fields. Use encryption, tight bucket policies and a governance layer like AWS Lake Formation for column and row level control. Keep data lineage so you can trace a prediction back to the files that fed it.

Model artifacts

Checkpoints, tokenizer like vocabularies for column types, and evaluation outputs also live on S3. Use a clear prefix per run, and store a manifest of the data snapshot used. A benchmark run, as described on /model/benchmarks, is easier to repeat when inputs and outputs are in versioned locations.

Practical guidance and best practices

Layout and naming

• Choose a stable, readable key scheme: zone/dataset/partition=value/file.
• Use zones such as raw, clean and curated, in separate buckets or prefixes.
• Partition by a column that queries filter on, usually a date. Avoid partitions with huge cardinality.
• Keep partitions from becoming tiny. Aim for files in a healthy size range, often tens to a few hundred megabytes, rather than thousands of kilobyte files.

File formats

• Use Parquet or ORC for analytics, with compression.
• Keep CSV and JSON only in the raw zone.
• Write a schema and keep it versioned.
• Avoid mixing different schemas in one prefix.

Security

• Turn on block public access at the account level.
• Use encryption at rest by default, and require encrypted transport.
• Prefer roles over long lived keys.
• Grant least privilege, scoped by prefix.
• Use access points for different consumer groups.
• Enable logging so you can audit access.
• Consider object lock for data that must not change.

Cost and performance

• Use lifecycle rules for old data and incomplete uploads.
• Compact small files with a scheduled job.
• Use the same region for storage and compute to avoid transfer cost and delay.
• Use byte range reads and column pruning.
• Spread request load across prefixes if you drive very high request rates.
• Use inventory reports instead of listing huge prefixes.

Reliability

• Turn on versioning for important buckets, plus a rule to expire old versions.
• Use replication for disaster recovery needs.
• Test restores, not only backups.
• Write data to a temporary prefix and publish by writing a small commit marker or by using a table format.

Table formats

If you need updates, deletes or concurrent writers, do not hand roll it. Use Apache Iceberg or Delta Lake. Both track files in metadata so readers see a consistent snapshot. See also Iceberg cataloging for how catalogs track the current table state.

Feeding models

• Export model-ready tables as Parquet with explicit types and nulls.
• Record the snapshot or version of the data used for each training or scoring run.
• Keep a small, fast sample prefix for development.
• Store results with the key columns so they can be joined back.

Common mistakes

1. Treating S3 like a file system: expecting cheap renames or in place edits. Renames are copies.
2. Too many small files: millions of tiny objects slow every engine and inflate request counts.
3. Over-partitioning: partitioning by a high cardinality column creates a maze of near empty folders.
4. Leaving buckets public by accident: block public access and review policies.
5. Storing only CSV: types, nulls and column order get lost, and scans read every column.
6. No lifecycle rules: old versions, temp files and failed uploads pile up.
7. Cross region compute: reading from one region while computing in another adds delay and transfer cost.
8. Using long lived access keys in code: use roles and short lived credentials.
9. Ignoring versioning side effects: with versioning on, deletes leave older versions behind unless a rule removes them.
10. Relying on listing for freshness: listing big prefixes in loops is slow. Use events, manifests or a table format.
11. No snapshot of training data: later you cannot say which files a model saw.
12. Writing results over inputs: always write outputs to a separate prefix so you can compare and roll back.

Frequently asked questions

Is Amazon S3 a database?

No. Amazon S3 is an object store that keeps whole files under keys. It has no tables, indexes or transactions of its own. Query engines and table formats add those features on top of the files it stores.

What is the difference between a bucket, an object and a key?

A bucket is the container, an object is the stored data with its metadata, and a key is the unique name of the object inside the bucket. The slashes in a key look like folders, but they are only part of the string, and a prefix is just a shared beginning of keys.

Is S3 strongly consistent?

Yes for the common operations. After a successful write, overwrite or delete, later reads and listings reflect the change. This simplified many data tools, but S3 still has no atomic multi file commit, so table formats or committers are used when several files must appear together.

Which file format should I store tables in?

For analytics, a columnar format such as Parquet or ORC is the usual choice because it compresses well, stores types and lets engines read only the needed columns. CSV is fine for raw landing but should be converted early. Table formats like Iceberg or Delta add transactions on top of Parquet files.

How do I query data stored in S3?

You use an engine that reads files from S3, such as Athena, Spark, Trino or a warehouse that supports external data. A catalog like the Glue Data Catalog stores table names, schemas and partitions so you can write SQL by table name instead of by file path.

Why are many small files a problem?

Each file needs at least one request and some metadata work, so thousands of tiny files make jobs slow and increase request volume. Compacting them into fewer, larger files, or using a table format with compaction, fixes the problem for most workloads.

How do I keep S3 data secure?

Block public access, encrypt data at rest, use roles with least privilege, scope permissions by prefix, and log access. For fine grained rules by column or row, add a governance layer such as Lake Formation on top of the catalog.

Can S3 hold data for machine learning models?

Yes. Training sets, feature tables, checkpoints and results are commonly stored there. For tabular models, keep tables in typed columnar files with explicit nulls, version the snapshots you train on, and place compute in the same region as the data.

What are storage classes for?

Storage classes let you match cost to access pattern. Hot data stays in the standard class, rarely read data moves to infrequent access or archive classes, and intelligent tiering can move data automatically. Always check retrieval time and minimum storage duration before moving data.

Does S3 support updates and deletes of single rows?

Not by itself, because objects are rewritten as a whole. Table formats such as Iceberg and Delta Lake implement row level changes by writing new files and updating metadata, which gives the effect of updates and deletes on top of S3.

Glossary

• Object storage: a storage model that handles whole objects addressed by key.
• Bucket: the named container for objects.
• Key: the unique identifier of an object in a bucket.
• Prefix: the leading portion of a key used to group objects.
• Multipart upload: a method that uploads a large object in parts.
• Byte range request: a read of only a section of an object.
• Partition pruning: skipping data folders that cannot match a filter.
• Column pruning: reading only the columns a query needs.
• Parquet: a columnar file format common in data lakes.
• Table format: metadata layer that gives files table behavior and transactions.
• Lifecycle rule: automatic action on objects based on age or other conditions.
• Versioning: keeping previous versions of an object after changes.
• Replication: automatic copying of objects to another bucket or region.
• Presigned URL: a temporary link that allows one specific action.
• Block public access: settings that prevent public exposure of buckets.
• Data lake: a store of raw and processed data in open formats.
• Landing zone: the first area where raw data arrives.
• Compaction: merging many small files into fewer large ones.
• Snapshot: a consistent view of a dataset at a point in time.
• Tabular foundation model: a model pre-trained to learn from many tables and apply that knowledge to new ones.

Summary

Amazon S3 is the durable, elastic storage base of most cloud data lakes. It stores objects under keys in buckets, supports byte range reads that suit columnar files, offers strong consistency, and provides lifecycle, replication and security controls. It does not offer transactions or row level updates, so open table formats and catalogs are added on top when those are needed.

For tabular foundation models, S3 is the place where training tables, context slices, results and checkpoints live. Typed columnar files preserve row and column structure, mixed types and explicit missing values. Versioned snapshots support continual learning and reproducible runs. Good layout, compaction, encryption and least privilege access keep the whole flow fast, safe and traceable. Explore the related pages on the AWS Glue Data Catalog, AWS Lake Formation, Amazon EMR and /model/benchmarks to see how the rest of the stack fits around it.

Related

It belongs to AWS data analytics in my stack, under Data engineering and lakehouse. Others in the same group: