AWS DataSync

Overview

AWS DataSync is a managed transfer service that copies data between storage systems and AWS storage services. It is built for moving large numbers of files quickly and safely, for example from an on-premises file server into Amazon S3. It handles scheduling, parallel transfer, retries, integrity checks and filtering, so you do not write and babysit copy scripts.

Most data projects start with a transfer problem. Before any model, query or pipeline can run, the files must be where the compute is. DataSync is a common way to bring file-based tables, exports and archives into a data lake. For teams working with tabular foundation models, it is the front door that gets raw tables into object storage so they can be cleaned, profiled and used. See /model/architecture for how such models read tables.

What it is

DataSync is an online data transfer service. It copies data over the network, either over the internet, over a private link, or through a virtual private cloud endpoint. It can run on a schedule or on demand and can keep a destination in step with a source by copying only what changed.

The building blocks

• Agent: a virtual machine or appliance you deploy near on-premises storage. It reads from or writes to local file systems and talks to the DataSync service. Transfers between some AWS services need no agent.
• Location: a description of a storage endpoint, such as an NFS export, an SMB share, an S3 bucket or an Amazon EFS file system. Every task has a source and a destination location.
• Task: a transfer definition. It names a source, a destination and options such as filters, verification mode, bandwidth limits and schedule.
• Task execution: one run of a task, with its own statistics and status.

What it is not

DataSync is not a database replication tool and not a streaming system. It moves files and objects. For change data from databases use Change data capture tools, and for event streams use Apache Kafka or Amazon MSK. It is also not a general backup product, although it can feed one.

An everyday analogy

Imagine moving house. You could carry boxes one by one in your own car, keep a list on paper, and hope nothing gets lost. DataSync is a professional moving company. A crew arrives with trucks, packs boxes in parallel, labels them, checks each item against your inventory on arrival, and comes back later to move whatever you added. You tell them which rooms to move and when.

Where it came from

DataSync was introduced by Amazon in 2018 as a service for moving data between on-premises systems and AWS. Before that, teams used tools such as rsync, robocopy, vendor appliances or physical devices for bulk loads. Each worked, but each demanded effort to run, monitor and verify at scale. The service packaged a parallel, verified transfer protocol with an agent and a managed control plane.

Over time DataSync added support for more storage types, including transfers between AWS services, to and from other clouds and object stores, and Enhanced mode options for higher throughput and larger file counts. The details of supported endpoints change often, so treat the official documentation as the source of truth for any specific combination.

Related services

AWS offers other ways to move data. Physical devices exist for very large offline transfers. Managed file transfer services handle protocols such as SFTP. Database migration services replicate databases. DataSync fills the space of large, repeatable file transfers over a network.

How it works

Step 1: Install and activate an agent

If your source is on-premises, you deploy an agent as a virtual machine on a hypervisor or as a hardware appliance, or in the cloud. You activate it, which links it to your account and region. The agent needs network access to your storage and to the DataSync endpoints. Analogy: the moving crew's truck arrives at your door and checks in with the office.

Step 2: Create locations

You define the source location, for example an NFS export at a fake host such as nas.example.internal, and the destination location, for example an S3 bucket and prefix. For S3 you also give an IAM role the service can assume to write objects.

Step 3: Create a task

You choose the options:

• Filters: include or exclude patterns, so you move only what you need.
• Transfer mode: copy only changed data or copy everything.
• Verification: check only transferred data, check all data at the end, or skip.
• Overwrite behaviour: replace changed files or keep existing ones.
• Metadata handling: preserve timestamps, permissions and ownership where the target supports it.
• Bandwidth limit: cap the network use so business traffic is not starved.
• Schedule: run hourly, daily, weekly or on demand.
• Logging: send task logs to CloudWatch.

Step 4: Run the task

At start, DataSync scans the source and destination to compare them, then prepares a list of what to transfer. It then moves data in parallel streams, compressing and encrypting in transit. The agent reads, the service transfers, and the destination writes.

Step 5: Verify

After copy, DataSync can compare checksums for transferred files. This catches silent corruption on the wire or at rest. Think of the movers ticking each box off against the inventory.

Step 6: Report and repeat

The task execution shows the files and bytes transferred, skipped and failed, along with throughput and duration. On a schedule, the next run copies only changes since the last, which makes incremental sync efficient.

Why incremental runs are cheap

The first run of a big task moves everything and can take long. Later runs compare metadata, such as size and modification time, and copy only new or changed files. If 2 percent of files change daily, the daily run is much lighter than the first. The scanning step still takes time on very large file counts, which is why DataSync has options aimed at huge file trees.

Networking options

DataSync can reach the service in several ways, and the right one depends on policy.

• Public endpoint: traffic goes over the internet, encrypted. Simple to set up.
• VPC endpoint: traffic stays on a private path from your network into AWS. Preferred for sensitive data.
• Private link or dedicated connection: if you have a direct connection to AWS, transfers can use it for steadier speed.
• Firewall rules: the agent needs outbound access to the service and inbound or outbound access to storage. Document the ports and hosts.

Security model in plain words

Three separate permissions are involved. The agent must be allowed to read the source storage, using credentials or an export rule. The service must be allowed to write to the destination, using an IAM role. People must be allowed to create and run tasks, using IAM user or role policies. Keeping these three apart means a person who can start a task cannot necessarily read the data, and the role that writes to S3 cannot be used for anything else.

What gets copied, and what does not

Key concepts and terms

A worked example

A team has about 40 TB of CSV exports on an NFS file server at nas.example.internal. They want them in S3 under a raw prefix, with a nightly sync of new files. All names are fake.

Step 1: Deploy and activate the agent

The team installs the agent virtual machine on a host in the data centre, gives it network access to the file server, and activates it using an activation key from the console. The command line equivalent returns an agent ARN.

aws datasync create-agent \
  --agent-name "dc1-agent-01" \
  --activation-key "EXAMPLE-ACTIVATION-KEY" \
  --vpc-endpoint-id vpce-0example000000000 \
  --subnet-arns arn:aws:ec2:us-east-1:111122223333:subnet/subnet-0example \
  --security-group-arns arn:aws:ec2:us-east-1:111122223333:security-group/sg-0example

Step 2: Create the source and destination locations

aws datasync create-location-nfs \
  --server-hostname nas.example.internal \
  --subdirectory /exports/finance \
  --on-prem-config AgentArns=arn:aws:datasync:us-east-1:111122223333:agent/agent-0example

aws datasync create-location-s3 \
  --s3-bucket-arn arn:aws:s3:::example-raw-bucket \
  --subdirectory /finance \
  --s3-config BucketAccessRoleArn=arn:aws:iam::111122223333:role/example-datasync-role

Step 3: Create the task

aws datasync create-task \
  --name "finance-exports-nightly" \
  --source-location-arn arn:aws:datasync:us-east-1:111122223333:location/loc-0source \
  --destination-location-arn arn:aws:datasync:us-east-1:111122223333:location/loc-0dest \
  --options VerifyMode=ONLY_FILES_TRANSFERRED,OverwriteMode=ALWAYS,TransferMode=CHANGED,PreserveDeletedFiles=PRESERVE,BytesPerSecond=-1 \
  --excludes FilterType=SIMPLE_PATTERN,Value="*.tmp|*/.snapshot/*" \
  --schedule ScheduleExpression="cron(0 2 * * ? *)"

Step 4: Start and watch

aws datasync start-task-execution \
  --task-arn arn:aws:datasync:us-east-1:111122223333:task/task-0example

aws datasync describe-task-execution \
  --task-execution-arn arn:aws:datasync:us-east-1:111122223333:task/task-0example/execution/exec-0example

The status moves through launching, preparing, transferring, verifying and success. The output shows estimated and transferred files and bytes.

A numeric walk-through

Suppose the 40 TB hold 80 million files. At an effective 400 MB per second, moving 40 TB takes about 40,000,000 MB divided by 400, which is 100,000 seconds, or a little under 28 hours. In practice throughput varies with file size, source speed and link capacity, so plan with a test run. Many small files move slower than a few large ones because each file has overhead. If 1 percent of files change each night, about 800,000 files move in later runs, which may take well under an hour once the scan completes.

Step 5: Make the data usable

When files land in S3, register them for querying. For example, create a table over the prefix and read it with Amazon Athena, or process it with EMR Serverless. Convert CSV to Parquet in a first processing step so later reads are cheap.

Handling special cases

Many small files

Millions of tiny files are the hardest case. Each file costs a few operations on the source, on the network and on the destination. Ways to cope:

• Use a task mode designed for large file counts if available.
• Split the tree into several tasks by folder, each with its own schedule.
• Bundle tiny files into archives at the source if the use case allows.
• Use a manifest to target only the files you need.

Very large files

Large files stream well. A single multi-terabyte file is limited by the single stream and the object size limits of the destination, so check those limits before the move.

Files that change during copy

If a file is being written while it is copied, the copy may be inconsistent. Pick a quiet window, copy from a snapshot of the source, or use a staging folder where producers finish files before they are moved.

Cross-account and cross-region moves

For moves between AWS accounts or regions, grant the role access to both sides and make sure the bucket policy allows the role. Test with a tiny prefix first.

Monitoring a running task

Useful numbers while a task runs:

Set an alarm if a scheduled execution does not start or if it fails. A quiet task is not always a healthy one.

Example: filters in practice

Filters use simple patterns separated by a pipe.

--includes FilterType=SIMPLE_PATTERN,Value="/finance/2025*|/finance/2026*"
--excludes FilterType=SIMPLE_PATTERN,Value="*.tmp|*.bak|*/Thumbs.db"

Include filters narrow the scope to the paths you list. Exclude filters then remove unwanted items inside that scope. Test filters on a small run before a full migration.

Strengths and limitations

Strengths

• Purpose-built for fast, parallel, verified file transfer.
• Managed control plane with scheduling, filters and reporting.
• Incremental sync after the first run.
• Encryption in transit and integrity checks.
• Fewer custom scripts to maintain and monitor.
• Works across on-premises, AWS and some other storage types.
• Integrates with CloudWatch for logs and metrics, and with IAM for access.

Limitations

• It copies files and objects. It does not replicate databases or streams.
• It needs an agent for on-premises sources, which you must host and update.
• Network bandwidth is the real ceiling. Slow links stay slow.
• Scanning very large trees takes time, even for small changes.
• File-system semantics, such as certain permissions or special files, may not map perfectly to object storage.
• Transformation is not its job. Changing formats happens in later steps.
• Charges apply per amount of data copied, so repeated full copies add up. Check the official pricing page, as this article gives no amounts.

How it compares with alternatives

Rule of thumb: use DataSync when the unit of data is a file and the job is large or repeated. Use database tools for tables inside databases, and streaming tools for continuous events.

Migration pattern in four phases

1. Assess. Count files and bytes, find owners, mark data to retire.
2. Pilot. Move one folder, verify, query it, and measure throughput.
3. Bulk and incremental. Run the main copy, then nightly syncs while users keep working.
4. Cutover. Freeze writes, run a final sync, verify, switch users, and keep the source read-only for a period.

Choosing destination storage

For analytics and models, S3 is almost always the target, because query engines and processing frameworks read it natively.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They deal with rows and columns, mixed types such as numbers, text and dates, missing values, and sometimes columns that have no helpful names. They are pre-trained on large sets of synthetic or real tables and can predict from labelled example rows given in context. They support tasks such as imputation, anomaly detection, continual learning and semantic or sector classification. See /model/research and /model/benchmarks.

DataSync does not touch the model. It solves the step before anything else: getting the tables to where compute can reach them.

Getting real tables into the lake

Real-world tables often live on file servers as CSV, Excel exports, Parquet files or database dumps. To evaluate a model on a company's data, or to build a pre-training collection of real tables, those files must move to cloud storage. DataSync can do that on a schedule, with verification, so that the lake copy matches the source.

Keeping context data fresh

In-context learning depends on example rows. If the source tables change daily, the context tables should as well. A scheduled DataSync task keeps the raw layer current, and a downstream job rebuilds the context table. This supports continual use of a model without retraining, as the model reads new examples rather than changing its weights.

Integrity matters for tables

A single corrupted file can shift column alignment or truncate rows. In a table, that may look like a burst of missing values or odd outliers. Task verification protects against transfer-level corruption, so any weirdness later is a data issue, not a copy issue. This also helps when comparing imputation or anomaly detection results across runs, because the inputs are identical.

Preserving structure and metadata

Folder names often carry meaning, such as a date or a region. Preserving the folder layout in S3 keeps that meaning, which later jobs can turn into partition columns. File timestamps can also help order incremental loads. Where column names are missing or cryptic, keep any accompanying data dictionary files in the same transfer so the context is not lost. A catalog such as Data catalog or the AWS Glue Data Catalog can then hold descriptions.

Governance and privacy

Raw tables can hold personal data. Plan to apply controls before wide use: restrict the raw bucket, use AWS Lake Formation permissions, and de-identify early with Data tokenization or Spark UDF de-identification. Record where each file came from for Data lineage.

Pre-training collections

For building a corpus of tables, DataSync can gather files from many internal or external locations into one bucket. Later jobs then deduplicate, filter and check for overlap with test sets. See Synthetic data generation for the complementary approach of generating tables.

A raw to curated flow

A realistic flow for table files looks like this:

1. DataSync lands CSV and Excel exports into a raw prefix, one folder per source system and date.
2. A trigger starts a processing job that reads new files only.
3. The job checks the schema against an expected contract, rejects or quarantines bad files, and writes Parquet to a curated prefix.
4. The job updates catalog tables and writes a profile of each column: type, null share, distinct count and sample values.
5. A context-table job picks labelled rows for in-context use and a score job writes predictions.

# sketch of a schema check on a newly landed file
import pandas as pd

expected = {"order_id": "object", "customer_id": "object", "amount": "float64"}
df = pd.read_csv("orders_2026-01-15.csv")

missing = [c for c in expected if c not in df.columns]
if missing:
    raise ValueError(f"missing columns: {missing}")

null_share = df.isna().mean().round(4).to_dict()
print("null share per column:", null_share)

This kind of check turns a file arrival into a trustworthy table, which is what a tabular model needs. If column names are missing in a file, the check can fall back to positions and a stored data dictionary.

Avoiding duplicate and stale context

Incremental syncs can deliver the same record in two files, for example a corrected export. A later job should deduplicate by key and keep the latest version. If duplicates reach the context table, the model may see the same row several times and overweight it.

Why a clean landing zone helps evaluation

When teams compare a model against baselines on their own tables, they need the same input for every method. A frozen copy in S3, created by a verified transfer and named by date, gives that stable base. It also supports repeat runs for audits.

Practical guidance and best practices

Plan before you copy

• Inventory the source: file count, total size, largest and smallest files, depth of folders.
• Decide the S3 layout: bucket, prefixes and naming. It is hard to change later.
• Decide what to exclude, such as temporary files and snapshot folders.
• Estimate time using link speed, then validate with a test on a sample.
• Agree an ownership and retention policy for the copy.

Deploy the agent well

• Place the agent close to the storage, with a fast network path.
• Give it enough CPU, memory and disk as recommended in the documentation.
• Use a private endpoint if traffic must not cross the public internet.
• Keep the agent updated and monitor its status.
• Use several agents for a very large source to increase parallelism.

Configure tasks sensibly

• Use filters to avoid copying junk.
• Choose changed-only transfer mode for recurring syncs.
• Use verification, at least for transferred files.
• Set a bandwidth limit during business hours if links are shared.
• Send logs to CloudWatch and set alerts on failure.
• Name tasks clearly, including source, destination and cadence.

Secure it

• Give the S3 access role only the bucket and prefix it needs.
• Turn on default encryption for the destination bucket.
• Use VPC endpoints and security groups to limit network paths.
• Keep activation keys out of source control.

Hand off to the next step

After the copy, trigger processing. An event or a schedule can start a job that converts CSV to Parquet, validates schemas with Data contract validation, and registers tables in the catalog. Orchestrate with Apache Airflow if there are many steps.

Test restore and cutover

If DataSync is part of a migration, plan a final sync window. Freeze writes at the source, run a last incremental sync, verify, and then switch users to the new location. Keep the source read-only for a while as a fallback.

Cost and time control

• Do the first full copy once, then rely on incremental runs.
• Exclude data you will not use before copying, not after.
• Compress or archive cold folders at the source if it reduces file counts.
• Use S3 lifecycle rules on the destination to move old raw data to cheaper storage classes.
• Review task execution history monthly and retire tasks that no longer serve a purpose.

Documentation to keep

Write down, for every task: the owner, the source path, the destination path, the schedule, the filters, the verification mode, the expected size and the contact for failures. When someone leaves, this note saves days of detective work.

Quick checklist before the first run

• The agent shows as online and can reach the source path.
• The destination bucket exists, is encrypted and blocks public access.
• The role can write to the exact prefix.
• Filters have been tested on a sample.
• Verification mode is chosen on purpose.
• Logging to CloudWatch is on.
• An alarm exists for failed executions.
• The team knows who to call if the task stalls.
• The downstream job knows where the new files will appear.
• A plan exists for the final cutover sync.

After the migration

Once users have switched, the work is not over. Review access to the new location, set retention and lifecycle rules, update documentation and runbooks, and decide when the old storage can be retired. Keep a record of the final counts of files and bytes at source and destination as proof that nothing was lost.

Common mistakes

• Skipping the inventory. A surprise of hundreds of millions of tiny files can make the transfer far slower than planned.
• Forgetting exclusions. Snapshot folders, caches and temporary files get copied and waste time and money.
• Not setting verification. Silent corruption is rare but costly to find later.
• Running everything through a single small agent. The agent becomes the bottleneck.
• Ignoring bandwidth sharing. A full-speed transfer can slow business traffic.
• Assuming file permissions carry over to S3. Object storage uses a different permission model.
• Copying in a layout that is hard to query. Plan prefixes for partitioning from the start.
• Using DataSync for database changes. Use database-oriented tools.
• No alerting. A failed nightly task can go unnoticed for weeks.
• Leaving raw personal data open. Lock down the landing bucket before the first run.

• Treating one successful run as proof of a complete migration. Compare counts and sizes at the end.

• Not testing the restore path. If the copy is a backup, practise reading it back.

• Changing the folder layout midway. Downstream jobs that rely on prefixes will break silently.
• Leaving old tasks running. Forgotten schedules keep copying data nobody uses.

Frequently asked questions

What is AWS DataSync used for?

AWS DataSync is used to move large amounts of file or object data between storage systems and AWS. Typical uses are migrating file servers to S3, sending recurring copies for backup or analytics, and moving data between AWS storage services. It adds scheduling, filtering, verification and monitoring on top of the raw copy.

Do I always need an agent?

No. An agent is needed when the source or destination is on-premises or in a place the service cannot reach directly. For transfers between some AWS storage services, such as S3 and EFS, the service can run without an agent. Check the documentation for the exact combinations your project needs.

How does DataSync verify data?

After transfer, DataSync can compare checksums of the source and destination data. You can choose to verify only the files that were transferred, to verify the whole destination, or to skip verification for speed. Verifying at least the transferred files is a good default for important data.

Is DataSync good for continuous replication?

It is good for scheduled and incremental file sync, such as hourly or daily runs. It is not a real-time replication system. For continuous change feeds from databases, use change data capture tools, and for event streams use a streaming platform.

How fast is DataSync?

Speed depends on your network link, the source storage, file sizes and the number of files. The service uses parallel streams and can reach high throughput, but a slow link remains slow, and huge numbers of tiny files transfer more slowly than a few big ones. Run a test on a sample to estimate real throughput.

Can DataSync transform my data?

No, its job is to copy and verify. Some metadata handling and filtering is available, but changes such as converting CSV to Parquet or cleaning columns belong in a later step using Spark, SQL or another processing tool.

Does DataSync encrypt data?

Yes, data is encrypted in transit between the agent and the service. At the destination, you control encryption at rest, for example with default bucket encryption in S3. Also use private network paths if your policy requires that traffic avoid the public internet.

How do I move data for machine learning on tables?

Use DataSync to land the raw files in S3, then convert them to a columnar format, register them in a catalog and profile them. From there, build context tables, test sets and feature tables. The transfer step simply ensures the right files are in the right place with verified integrity.

Can I schedule and automate tasks?

Yes. You can attach a schedule to a task and run it hourly, daily or weekly. You can also start executions from code or from an orchestrator. Pair this with logging and alarms so failures are noticed quickly.

What happens if a transfer fails halfway?

DataSync retries and reports files that failed. A later execution of the same task picks up where the comparison shows differences, so you rarely need to restart from zero. Check the task logs for the cause, such as a permission problem or a file that changed during copy.

Glossary

• Agent: software that connects local storage to the DataSync service.
• Bandwidth throttle: a limit on network use.
• Checksum: a short value computed from data to detect changes or corruption.
• Cutover: the moment users switch from the old location to the new one.
• Filter: a pattern that includes or excludes paths.
• Incremental sync: copying only what changed since the last run.
• Location: a storage endpoint known to the service.
• Manifest: an explicit list of files to move.
• NFS: a network file system protocol common on Linux and storage appliances.
• SMB: a network file sharing protocol common on Windows.
• Task: a saved transfer configuration.
• Task execution: one run of a task.
• Throughput: amount of data moved per second.
• VPC endpoint: a private connection from a virtual network to a service.

Summary

AWS DataSync is a managed service for copying large volumes of files between on-premises storage, AWS storage and other locations. It uses an agent, locations and tasks, moves data in parallel, verifies integrity, supports filters and schedules, and syncs only changes after the first run. It is the right choice when the unit of data is a file and the job is big or recurring.

For tabular foundation models, DataSync gets raw tables into the lake reliably so they can be profiled, cleaned, sampled and governed. It keeps context data fresh for in-context use, protects against silent corruption that could look like missing values or outliers, and preserves folder structure and metadata that later become partitions and column descriptions. Plan the layout, verify the data, secure the landing zone and hand off to processing jobs.

Related

It belongs to AWS data analytics in my stack, under Data engineering and lakehouse. Others in the same group: