Amazon RDS

Overview

Amazon RDS (Relational Database Service) is a managed service that runs relational databases in the cloud. It supports engines such as PostgreSQL, MySQL, MariaDB, Oracle and SQL Server, and the provider handles routine work: provisioning servers, patching, automated backups, monitoring and failover. You still design schemas, write queries and tune performance, but you do not manage the operating system or the hardware.

For data teams, RDS is a common home for operational tables: orders, customers, events, features and model outputs. Those tables are the raw material for tabular machine learning, including tabular foundation models that learn from rows and columns directly. This page explains how RDS works, how to use it well and how it fits around model workflows. Related pages: PostgreSQL, pgvector, Amazon S3, Terraform, HashiCorp Vault and Snowflake.

What it is

RDS is a family of managed database instances. You choose an engine, a version, an instance size and a storage type. The service creates a database endpoint, which is a network name your applications connect to using the standard protocol of the engine. Because it speaks the real engine protocol, existing drivers, tools and SQL knowledge carry over.

What the service takes care of

• Provisioning. Launch an instance in minutes.
• Patching. Minor version updates in maintenance windows.
• Backups. Automated daily snapshots plus transaction logs for point-in-time recovery.
• High availability. An optional standby copy in another availability zone with automatic failover.
• Read scaling. Read replicas for reporting and analytics.
• Monitoring. Metrics for CPU, memory, storage, connections and latency.
• Encryption. Storage encryption with managed keys and encrypted connections.
• Access control. Network isolation and identity integration.

What you still own

• schema design and indexing
• query tuning
• capacity planning and cost
• user and role design inside the database
• data lifecycle and retention
• application behaviour on failover, such as reconnecting

Engines in brief

Exact versions and features change over time, so check the current documentation before planning.

Where it came from

Running a database used to mean racking servers, installing software, writing backup scripts and waiting for midnight maintenance. As cloud computing matured, providers offered virtual machines, and teams installed databases on them. That removed the hardware but not the toil. Backups, patching and failover were still the team's job.

The managed database idea

RDS launched in the late 2000s with the idea that a database is a service, not a server. The provider would automate the repetitive tasks and expose the usual engine on a network address. Over the years it added more engines, Multi-AZ deployments for failover, read replicas, encryption, performance monitoring tools and serverless-style scaling options in some variants.

Why it spread

• Small teams could run production databases without a dedicated administrator.
• Larger firms could standardize backups, security settings and upgrades.
• Pay-as-you-go pricing and quick provisioning suited experiments and growth.
• Compatibility with open source engines reduced lock-in fears.

The wider trend

RDS sits in a family of managed data services: object storage, warehouses, streaming and search. A common modern design uses RDS for transactional work, object storage for the raw lake and a warehouse or lakehouse for analytics, with pipelines moving data between them.

How it works

Think of renting a fully serviced kitchen in a food hall. You bring your recipes and cook. The hall provides ovens, cleaning, fire safety and a spare kitchen next door if yours has a problem. You do not fix the ovens, but you are still responsible for the food.

Step 1: Create an instance

You pick an engine and version, an instance class (CPU and memory), storage size and type, and a network location (a virtual private cloud and subnets). You set a master username and choose how it is authenticated. You decide whether the instance is public or private; private is the safer default.

Step 2: Storage and durability

Data lives on network-attached block storage that is replicated within a zone. You choose a storage type that balances cost and input/output speed. Storage can grow, and in many cases can scale automatically up to a limit you set.

Step 3: Connect

Your application connects to the endpoint with a standard driver. Security groups, which act like firewalls, decide which clients can reach the port. TLS can encrypt the connection.

Step 4: Backups and point-in-time recovery

The service takes automated backups on a schedule, and it keeps transaction logs. This lets you restore to any second within the retention period. Restoring creates a new instance. Think of it as having both daily photos and a continuous video of your database.

Step 5: Multi-AZ failover

In a Multi-AZ deployment, a standby copy runs in another availability zone. The primary replicates data to it. If the primary fails, the service promotes the standby and updates the endpoint name. Applications reconnect, usually within a minute or two. The standby is for availability, not for reads, in the classic design. Some newer designs offer readable standbys.

Step 6: Read replicas

A read replica is a separate instance that receives changes from the primary, usually asynchronously. You send reporting queries and training data extracts to the replica, so they do not slow down the main application. Replicas can lag behind, so they may show slightly old data.

Step 7: Maintenance

The service applies patches during a maintenance window you choose. Major version upgrades are planned events that you control. Testing on a copy first is wise.

Step 8: Monitoring

Metrics such as CPU use, free memory, connections, disk queue depth and replica lag are available. Extra tools show which queries are slowest and what they wait on. Alarms notify you when thresholds are crossed.

Step 9: Security

• Network isolation with private subnets.
• Encryption at rest using managed keys.
• Encryption in transit with TLS.
• Identity integration so that applications can use short-lived tokens instead of static passwords, for engines that support it.
• Audit logging options depending on the engine.
• Secrets can be rotated by an external manager such as HashiCorp Vault or a cloud secrets service.

An everyday analogy for each feature

• Automated backups: a security camera recording the shop all day.
• Multi-AZ: a spare shop in the next town that copies your stock every minute.
• Read replica: a photocopy of the price list for customers to browse, so the cashier is not interrupted.
• Parameter groups: a settings card for the engine that you can swap.
• Snapshots: a labelled photo of the shop at a moment you choose, kept until you delete it.

Storage types and when to choose them

A simple rule: start with general purpose storage, watch the disk queue depth and latency metrics, and move to provisioned speed only when the metrics prove the need.

Sizing a database by working backwards

1. Estimate the data size today and growth per month.
2. Estimate the working set, the part of the data touched often. If it fits in memory, reads are fast.
3. Estimate peak queries per second and typical query cost.
4. Choose an instance class with memory close to the working set.
5. Load test with realistic data and queries, not an empty database.

As a rough example, if the hot data is 20 GB and the instance has 32 GB of memory, most reads will hit the cache. If the hot data is 200 GB, expect disk reads and plan for faster storage or a larger class.

Monitoring signals worth alarming on

• CPU above a high threshold for a sustained period.
• Free storage below a safety margin.
• Freeable memory dropping steadily, which can signal a leak or a too-small class.
• Connection count near the limit.
• Replica lag above your tolerance.
• Rising disk queue depth or read latency.
• Failed login bursts.

Cost levers

• Right-size instance classes after observing real load.
• Use reserved or committed pricing where usage is steady and the contract suits you.
• Stop or delete development and test instances outside working hours.
• Delete old manual snapshots that no one needs.
• Compress and archive old rows to object storage.
• Reduce replicas that sit idle.

Migration paths

Always rehearse a migration on a copy, measure the cutover time and prepare a rollback plan.

Data lifecycle for model work

Keep the operational database lean. Move old rows to object storage in columnar files, partitioned by date. Keep a catalog of which snapshot fed which model version. When a regulation asks you to delete a person's data, you must be able to find it in the database, the replicas, the backups within their retention and the exported training copies.

Key concepts and terms

A worked example

This example sets up a small PostgreSQL instance with infrastructure code, then reads training data from a replica. All names and hosts are fake.

Infrastructure as code

resource "aws_db_instance" "orders" {
  identifier              = "orders-db"
  engine                  = "postgres"
  engine_version          = "16"
  instance_class          = "db.m6g.large"
  allocated_storage       = 100
  max_allocated_storage   = 500
  storage_encrypted       = true
  multi_az                = true
  db_name                 = "orders"
  username                = "app_admin"
  manage_master_user_password = true
  backup_retention_period = 7
  deletion_protection     = true
  publicly_accessible     = false
  vpc_security_group_ids  = [aws_security_group.db.id]
  db_subnet_group_name    = aws_db_subnet_group.private.name
}

resource "aws_db_instance" "orders_replica" {
  identifier          = "orders-db-replica"
  replicate_source_db = aws_db_instance.orders.identifier
  instance_class      = "db.m6g.large"
  publicly_accessible = false
}

Version numbers and instance class names are examples. Check what is available in your region. See Terraform for the tool used above.

Notes on the sketch

• multi_az gives automatic failover.
• storage_encrypted turns on encryption at rest.
• manage_master_user_password lets the service keep the master password in a managed secret instead of in code.
• deletion_protection prevents accidental deletion.
• The replica is for reporting and for pulling training data.

Reading data for modelling

import pandas as pd
import sqlalchemy as sa

engine = sa.create_engine(
    "postgresql+psycopg://reader@orders-db-replica.example.internal:5432/orders",
    connect_args={"sslmode": "require"},
)

query = """
SELECT order_id, customer_segment, items, order_total, country, returned
FROM orders
WHERE order_date >= current_date - interval '90 days'
"""

df = pd.read_sql(query, engine)
print(df.shape)

The password would come from a secrets service or token-based login, not from the URL. A read replica is used so the extraction does not disturb the live application.

A numeric walk-through of backup retention

Suppose your database is 100 GB, the retention period is 7 days and daily changes are about 5 GB. The service stores the base backup and the logs needed to replay changes. Roughly, you would keep one full copy plus 7 days of logs, so extra storage is on the order of the daily change times the retention period, here about 35 GB, before compression. Your actual numbers will differ, but this shows why longer retention increases cost in proportion to change rate, not to total size.

A numeric walk-through of failover time

If a failover takes 60 to 120 seconds, and your application retries connections every 5 seconds for up to 3 minutes, the app will recover without human help. If it retries only once and gives up, the same failover becomes an outage. The database was healthy; the client was not ready.

Checking replica lag

-- On a PostgreSQL replica
SELECT now() - pg_last_xact_replay_timestamp() AS replica_lag;

If lag is a few seconds, a training extract is nearly current. If it is minutes, wait or read from the primary during a quiet period.

Strengths and limitations

Strengths

• Less operational work. Backups, patching and failover are built in.
• Familiar engines. Existing SQL skills, drivers and tools work.
• Availability options. Multi-AZ and replicas improve resilience and read capacity.
• Security features. Encryption, private networking and identity integration.
• Fast provisioning. Test and staging environments are quick to create from snapshots.
• Integration. Works with monitoring, secrets, networking and infrastructure-as-code tools.

Limitations

• Limited OS and superuser access. You cannot install arbitrary software or use every extension. Only supported extensions are available.
• Cost. Always-on instances, Multi-AZ, provisioned IOPS and replicas add up.
• Vertical scaling limits. Relational engines scale writes mostly by using a bigger instance, not by adding many writers.
• Not a warehouse. Large analytical scans can hurt an operational database. Use replicas or move data to an analytical system.
• Maintenance events. Patches and failovers can cause brief interruptions.
• Engine differences. Features differ between engines and versions.
• Lock-in at the edges. The engine is portable, but surrounding features such as monitoring and identity integration are not.
• Shared responsibility. Security misconfiguration, such as a public endpoint with a weak password, is still the customer's risk.

How it compares with alternatives

A common design uses RDS for operational data, copies changes to a lakehouse for analytics and trains models from the lakehouse copy.

Role in tabular foundation models

RDS is a source and a sink for tabular work. Tabular foundation models learn from raw tables: rows, columns, mixed numeric and categorical types, missing values and sometimes no meaningful column names. They are pre-trained on synthetic or real tables and use in-context learning, where a handful of labelled rows supplied at prediction time condition the model. See /model/architecture and /model/research. A relational database fits in these ways.

Natural fit for the data shape

A relational table is exactly the structure these models expect: one row per entity or event, one column per attribute. Types such as integers, floats, text, booleans and timestamps map to the mixed types that tabular models must handle. NULL in SQL is missing data, which these models treat as a signal rather than an error.

Where the rows come from

Operational tables in RDS are often the real tables used for evaluation and fine-tuning. A typical flow reads a snapshot or replica, joins a few tables into a single wide table, and passes it to the model.

Feeding in-context learning

At inference, a service may select a few recent labelled rows, for example the last 500 resolved cases, and give them to the model as context along with the new rows to score. A database query selects those rows. Good indexes and a replica keep that query fast. The rule of thumb: keep context queries simple, bounded by LIMIT and run against a replica.

Writing predictions back

Predictions, imputed values, anomaly scores and sector or semantic labels can be written back to RDS in a results table. Use a separate schema or table for model outputs, with the model version and a timestamp. This keeps predictions auditable and avoids overwriting source facts.

Imputation and missing values

SQL NULLs flow naturally into model inputs. For imputation tasks, the model fills missing cells, and the results can be stored in a second table or a view. Keep the original NULL in the source so you can always tell what was observed and what was inferred.

Anomaly detection

A model can score rows from a table, such as transactions, and write an anomaly score. Triggers or scheduled jobs can then flag rows above a threshold. Database logs and audit tables, which are also tabular, are good candidates for anomaly detection on the database itself.

Continual learning

New rows arrive constantly. Continual learning on tabular models needs a reliable way to get the delta since the last update. Use an updated_at column with an index, or logical replication and change data capture to a lake. Record the watermark of the last row processed so each update sees new data exactly once.

Vectors and embeddings

With PostgreSQL-compatible engines, an extension such as pgvector can store embeddings next to rows, if the extension is supported in your version. This helps when text columns are embedded and searched. Check engine support before designing around it.

Security and de-identification

Tables going to training should be minimized and de-identified. Views can hide columns, and jobs such as Spark UDF de-identification can create protected copies. Credentials should be short-lived, as described in HashiCorp Vault, and keys managed carefully, as in Thales.

A practical pipeline

1. Application writes to the primary RDS instance.
2. Change data is copied to object storage such as Amazon S3 in a columnar format.
3. A job de-identifies and joins tables.
4. A tabular model trains, adapts or scores.
5. Predictions go back to a results table in RDS or to a warehouse.

This description is generic and does not claim that any product has a built-in connector for any given model. Benchmark thinking for such models is covered at /model/benchmarks.

Practical guidance and best practices

Choose the engine on purpose

If you have no strong reason, a mainstream open source engine such as PostgreSQL is a good default. Pick the engine your team knows, your tools support and your licences allow.

Use Multi-AZ for production

A standby in another zone gives automatic failover. Test a failover in staging so you know how your application behaves.

Keep instances private

Place instances in private subnets, restrict security groups to known clients and avoid public access. Use a bastion or a private network path for administration.

Encrypt everything

Turn on storage encryption when creating the instance, since changing it later requires a copy. Require TLS for connections.

Do not hard-code passwords

Use a managed secret, short-lived tokens or a secrets service. Rotate regularly.

Right-size, then review

Start with a modest size and watch CPU, memory, connections and storage. Resize when metrics show sustained pressure. Over-provisioning wastes money, and under-provisioning causes slow queries.

Index for your queries

Look at slow queries and add indexes where they help. Remember each index slows writes and uses space. Remove unused indexes.

Use connection pooling

Databases handle a limited number of connections. Many short-lived clients, such as functions or notebooks, can exhaust them. A pooler or proxy reduces the problem.

Separate workloads

Send reporting, training extracts and data science to read replicas. Keep the primary for the application.

Set backup retention and test restores

Choose a retention period that matches your recovery needs. Restore to a test instance now and then to check that backups work and to measure the time it takes.

Tag and automate

Tag instances with owner and purpose. Define them in code, so environments are repeatable.

Plan upgrades

Track end-of-support dates. Test major upgrades on a restored copy and measure query plans before and after.

Watch costs

Delete old snapshots, stop idle development instances and review replica count. Storage and IO choices can dominate the bill.

Prepare for failover in code

Clients should use the endpoint name, not an IP address, and retry with backoff. Keep transactions short so they roll back cleanly.

Common mistakes

• Public endpoint with a weak password. This is a frequent cause of breaches.
• Running heavy analytics on the primary. Long scans slow the application.
• No tested restore. A backup is only proven when restored.
• Single zone for production. A zone issue then means an outage.
• Ignoring connection limits. Applications scale up and exhaust connections.
• Forgetting storage growth. Disks that fill up cause outages, so set alarms and automatic scaling limits.
• Using IP addresses instead of the endpoint name. Failover changes the underlying address.
• Treating the replica as always fresh. Replica lag can mislead reports and training sets.
• Skipping parameter tuning. Defaults are generic and may not fit your workload.
• Large unbounded queries for model context. Always bound them.
• Overwriting source data with predictions. Keep model outputs in separate tables with versions.
• Neglecting minor patches. Security fixes need timely application.
• Letting dev instances run forever. Cost creeps up unnoticed.

Frequently asked questions

What is Amazon RDS?

Amazon RDS is a managed relational database service that runs engines such as PostgreSQL, MySQL, MariaDB, Oracle and SQL Server. The provider automates provisioning, patching, backups and failover, while you manage schemas, queries and performance. You connect with ordinary database drivers.

What is the difference between Multi-AZ and a read replica?

Multi-AZ keeps a standby copy in another availability zone for automatic failover and is mainly about availability. A read replica is an extra copy that serves read-only queries to spread load and is mainly about read scaling. Replicas usually lag slightly behind, and failover to a replica is a separate, more manual choice.

Is RDS the same as running PostgreSQL myself?

The engine is the same, but the operations differ. With RDS, the provider handles the server, backups and failover, and you cannot log into the operating system or install any extension. With self-managed PostgreSQL, you control everything and take on all the operational work. See PostgreSQL for the engine itself.

How do backups and point-in-time recovery work?

The service takes automated daily snapshots and also keeps transaction logs. To recover, you restore to a chosen moment within the retention window, and a new instance is created with the data as it was then. Manual snapshots can be kept longer. Always test a restore to learn how long it takes.

Can I train machine learning models on data in RDS?

Yes. Extract tables from a read replica or copy them to object storage or a lakehouse, then train on that copy. Avoid heavy scans on the primary. De-identify sensitive columns before exporting, and keep a record of which snapshot was used for each model version.

Does RDS work with tabular foundation models?

RDS does not include models, but its tables are the kind of input these models use. A pipeline selects rows with SQL, passes them as context or as rows to score, and writes predictions back to a results table. The model handles mixed types and missing values, which map naturally to SQL columns and NULLs.

How should I store model predictions in the database?

Write them to a dedicated table with the entity id, the prediction, a confidence value if available, the model version and a timestamp. Do not overwrite source columns. This keeps history clear and lets you compare model versions.

How do I keep an RDS instance secure?

Place it in private subnets, limit security group rules, enable encryption at rest and TLS, use short-lived or managed credentials, restrict database roles to least privilege and enable audit logging where the engine supports it. Review access regularly.

When is RDS the wrong choice?

It is a weaker fit for very large analytical scans, for workloads needing massive horizontal write scaling, for engines or extensions the service does not support, and for cases needing full operating system control. A warehouse, lakehouse or a self-managed cluster may suit those better.

How does RDS handle failover from the application's view?

The endpoint name stays the same but points to the new primary after promotion. Existing connections drop, and the application must reconnect. Use retries with backoff and short transactions so the application recovers without manual help.

Glossary

• Availability zone: A separate physical location within a region.
• Backup retention: How long automated backups are kept.
• Connection pool: A cache of database connections shared by clients.
• Endpoint: The network name used to connect to the database.
• Failover: Switching to a standby after a primary failure.
• IOPS: Input/output operations per second.
• Instance class: The size category of a database instance.
• Logical replication: Replication of data changes at the row level.
• Maintenance window: A scheduled period for patching.
• Multi-AZ: A deployment with a standby in another zone.
• NULL: The SQL marker for a missing value.
• Parameter group: A named set of engine settings.
• Point-in-time recovery: Restoring to a chosen moment.
• Primary: The instance that accepts writes.
• Read replica: A copy that serves reads.
• Replica lag: The delay between primary and replica.
• Security group: A virtual firewall.
• Snapshot: A saved backup of the database storage.
• Standby: A passive copy ready to take over.
• Subnet group: The set of subnets an instance may use.

Summary

Amazon RDS is a managed service for relational databases such as PostgreSQL, MySQL, Oracle and SQL Server. It automates provisioning, patching, backups, monitoring and failover, while leaving schema design, indexing and tuning to you. Multi-AZ gives resilience, read replicas give read capacity, and private networking, encryption and identity integration give security. For tabular foundation models, RDS holds the row-and-column tables that models learn from or score, supplies recent labelled rows for in-context learning, and stores predictions, imputed values and anomaly scores. Good practice is to keep instances private and encrypted, use replicas for training extracts, de-identify sensitive columns, write model outputs to separate versioned tables and test restores and failovers. For the model side, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Databases and sync in my stack, under Data engineering and lakehouse. Others in the same group: