AWS CDK

Overview

AWS CDK, the Cloud Development Kit, is a framework for defining AWS infrastructure in a general purpose programming language such as TypeScript or Python. You write classes and function calls. The CDK turns them into CloudFormation templates, and CloudFormation creates the real resources. The idea is that infrastructure gets the same tools as application code: loops, functions, types, tests and package sharing.

Data and machine learning teams on AWS use the CDK to define storage buckets, queues, functions, containers, clusters and permissions in one place. A data platform that serves tabular models can then be created, reviewed and rebuilt from source, with high level building blocks that hide much of the repetitive wiring.

What it is

The CDK has three parts: a library of building blocks called constructs, a command line tool, and a process called synthesis that outputs templates.

• Construct library. Classes that represent AWS resources and patterns.
• CLI. Commands such as cdk init, cdk synth, cdk diff, cdk deploy and cdk destroy.
• Synthesis. Running your program to produce CloudFormation and asset files.

An everyday analogy

CloudFormation is like a very detailed blueprint written by hand. The CDK is like architecture software: you place a "kitchen" component and the software draws the pipes, outlets and cabinets for you. You can still edit the raw drawing, but most of the time you work with the higher level parts.

Supported languages

The CDK core is written in TypeScript and exposed to other languages such as Python, Java, C# and Go through a translation layer. The concepts are identical across languages. Examples here use Python and TypeScript.

Imperative code, declarative result

Your program runs once at synthesis time. It builds an in-memory tree of constructs, then writes out a declarative template. The deployed result is still declarative, with CloudFormation tracking state and rolling back failed changes. Your code is a template generator, not a runtime script.

Where it came from

CloudFormation, the AWS template service, appeared in the early 2010s. Teams wrote very long JSON or YAML files and found them hard to reuse. Third party tools such as Terraform offered another language and a multi cloud approach. See Terraform.

AWS introduced the CDK in the late 2010s to let developers use real languages, and it became generally available in 2019. A major rewrite called version 2 later merged the library into a single package and simplified dependency handling. A sister project applied the same construct model to Kubernetes manifests, and another to Terraform configurations, so the construct idea spread beyond AWS.

The design draws on a long programming tradition of composition: small parts assembled into bigger parts, shared through package managers.

How it works

Step 1: Define an app and stacks

An app is the root. Inside it are stacks, which are units of deployment that map to CloudFormation stacks. Inside stacks are constructs.

Step 2: Use constructs at three levels

Higher levels give you convenience; lower levels give you control. You can mix them, and you can reach into an L2 construct to change an underlying L1 property when needed.

Step 3: Synthesise

cdk synth runs your code and produces a cloud assembly: templates plus any assets, such as zipped function code or container image definitions.

Step 4: Diff

cdk diff compares the synthesised template to the deployed stack. It shows what will be added, changed or removed, including permission changes. This is the CDK version of a plan.

Step 5: Bootstrap and deploy

A one time cdk bootstrap per account and region creates helper resources, such as a bucket for assets and roles for deployment. cdk deploy uploads assets and asks CloudFormation to create or update the stacks. CloudFormation handles ordering, parallelism, and rollback on failure.

Step 6: Iterate

You edit code and repeat. cdk destroy removes the stacks. Stateful resources can be given removal policies so that a destroy does not delete valuable data.

Grants and permissions

One of the most loved features is the grant methods. Instead of writing a policy by hand, you write bucket.grant_read(role) and the CDK adds the least set of permissions needed. This saves time and reduces mistakes.

Context and environments

Stacks can be tied to an account and region. Context values and lookups let the code ask about existing resources, such as a VPC by tag. Lookups are cached in a file so synthesis stays predictable.

Key concepts and terms

A worked example

This example creates a storage bucket for curated tables, a queue that receives file notifications, and a function that validates arriving files. It is a small slice of a tabular data intake path.

Python version

from aws_cdk import (
    App, Stack, Duration, RemovalPolicy,
    aws_s3 as s3,
    aws_sqs as sqs,
    aws_lambda as _lambda,
    aws_s3_notifications as s3n,
)
from constructs import Construct


class IntakeStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        bucket = s3.Bucket(
            self, "IncomingTables",
            versioned=True,
            encryption=s3.BucketEncryption.S3_MANAGED,
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            removal_policy=RemovalPolicy.RETAIN,
        )

        dead_letter = sqs.Queue(self, "FailedFiles")

        validator = _lambda.Function(
            self, "Validator",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="app.handler",
            code=_lambda.Code.from_asset("lambda/validator"),
            timeout=Duration.seconds(60),
            dead_letter_queue=dead_letter,
        )

        bucket.grant_read(validator)
        bucket.add_event_notification(
            s3.EventType.OBJECT_CREATED,
            s3n.LambdaDestination(validator),
            s3.NotificationKeyFilter(suffix=".parquet"),
        )


app = App()
IntakeStack(app, "IntakeStack")
app.synth()

TypeScript version of the same idea

import { App, Stack, StackProps, Duration, RemovalPolicy } from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';

class IntakeStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const bucket = new s3.Bucket(this, 'IncomingTables', {
      versioned: true,
      removalPolicy: RemovalPolicy.RETAIN,
    });

    const validator = new lambda.Function(this, 'Validator', {
      runtime: lambda.Runtime.PYTHON_3_12,
      handler: 'app.handler',
      code: lambda.Code.fromAsset('lambda/validator'),
      timeout: Duration.seconds(60),
    });

    bucket.grantRead(validator);
  }
}

const app = new App();
new IntakeStack(app, 'IntakeStack');

The validator function

import json

REQUIRED = {"customer_id", "event_time", "amount"}

def handler(event, context):
    # In a real system this would read the file header or schema.
    # Here we only show the shape of a check.
    records = event.get("Records", [])
    results = []
    for rec in records:
        key = rec["s3"]["object"]["key"]
        results.append({"key": key, "checked": True})
    return {"statusCode": 200, "body": json.dumps(results)}

Commands

cdk bootstrap
cdk synth
cdk diff
cdk deploy IntakeStack

What you gained

Roughly forty lines of code produced the bucket, encryption, public access block, queue, function, role, policy and event wiring. In raw CloudFormation this would be several hundred lines. The grant_read call produced a scoped policy. The RETAIN policy protects the data on stack deletion.

Looping and reuse

Because it is code, you can generate one bucket per data domain.

for domain in ["sales", "support", "sensors"]:
    s3.Bucket(self, f"{domain.title()}Tables", versioned=True)

Strengths and limitations

Strengths

• Real languages: loops, conditions, functions, classes, types and IDE help.
• High level constructs with sensible defaults reduce boilerplate.
• Grant methods produce least privilege policies with little effort.
• Built on CloudFormation, so deployments are managed, ordered and roll back on failure.
• Easy to test, since constructs can be asserted against with unit tests.
• Shareable libraries: teams package approved patterns for others to reuse.
• cdk diff surfaces security relevant changes before deploy.

Limitations

• AWS only for the main CDK. Multi cloud needs other tools.
• Inherits CloudFormation limits, such as resource counts per stack and slow updates for some resources.
• The synthesis step can hide what is generated; you must read the output sometimes.
• Too much programming freedom can lead to hard to read infrastructure code.
• Some services have only L1 constructs, so you write more detail yourself.
• Logical ID changes from renaming a construct can replace a resource.
• Upgrades of the library can alter generated templates, so review diffs.
• Not a tool for the data inside tables; use data tools for schema and quality work.

How it compares with alternatives

If your estate is AWS only and your team prefers a familiar language, the CDK is a strong fit. If you manage many providers, a neutral tool may be simpler.

Role in tabular foundation models

A tabular foundation model is trained to understand raw tables: rows and columns, numeric and categorical and text values, missing entries, and sometimes columns with no useful names. It is pre-trained on many tables, including synthetic ones, and at use time it can adapt to a new table from a few examples using in-context learning. The CDK does not touch that learning. It builds the AWS platform that stores the data, runs the jobs and controls access.

Intake of raw tables

Raw tables arrive as files, database exports or streams. The CDK can define the landing buckets, notification queues, validation functions and dead letter queues, as in the worked example. That intake layer is where checks on column count, types and missing rates should run, before a model ever sees the table. See Schema-drift detection and Data contract validation.

Catalogs and warehouses

Tables are easier to find and govern when registered in a catalog. The CDK has constructs for catalog and ETL services, so a team can declare databases, crawlers and jobs. See AWS Glue Data Catalog and Data catalog. Open table formats such as Delta Lake and Iceberg rely on this kind of storage layout; see Iceberg cataloging.

Compute for training and inference

Training or fine tuning needs accelerator instances; inference needs scalable endpoints. The CDK can declare container services, clusters, batch environments and managed machine learning resources. High level patterns reduce the wiring for load balanced model servers. See Docker and Kubernetes.

Batch tasks that use the model

Imputation of missing values, anomaly flagging and sector or semantic classification of whole tables are often scheduled batch jobs. The CDK defines the schedules, state machines, queues and permissions for them. See Imputation networks and Anomaly detection.

Governance of sensitive data

Grant methods and aspects help enforce rules such as "all buckets are encrypted" or "no public access". An aspect can walk through every construct and flag violations at synthesis time, which is useful for data that includes personal information.

Repeatable test environments for continual learning

When a model is updated over time, you compare a candidate with the current one on the same inputs. Defining the whole stack in code lets you create an identical, short lived environment for each candidate and delete it afterwards. See MLflow for tracking versions, and /model/research, /model/architecture and /model/benchmarks for model context.

Honest boundaries

The CDK is infrastructure code. It does not understand tables, and its generated templates are only as good as the design behind them. Model training logic stays in your training code.

Practical guidance and best practices

Structure the app

• One stack per lifecycle: keep stateful resources (data stores) apart from stateless ones (functions).
• Split large systems into several stacks to stay within resource limits.
• Put shared constructs in a library and version them.
• Keep environment specific values in configuration, not scattered through code.

Protect data

• Set removal policies to retain for buckets and databases.
• Enable versioning and encryption.
• Block public access by default.
• Turn on termination protection for critical stacks.

Name things carefully

• Let the CDK generate resource names where you can; fixed names block replacement.
• Keep construct IDs stable. Renaming an ID changes the logical ID and may replace the resource.
• Use tags consistently for owner and purpose.

Test the code

• Use assertions on synthesised templates to check key properties.
• Add snapshot tests for sensitive stacks, and review snapshot changes.
• Run cdk diff in pull requests.

from aws_cdk import App, assertions
from intake.intake_stack import IntakeStack

def test_bucket_is_encrypted():
    app = App()
    stack = IntakeStack(app, "T")
    template = assertions.Template.from_stack(stack)
    template.has_resource_properties("AWS::S3::Bucket", {
        "BucketEncryption": assertions.Match.any_value()
    })

Automate delivery

• Run synth, tests and diff in a pipeline. See GitLab CI/CD, GitHub Actions and Jenkins.
• Use an approval step before production.
• Use a dedicated deployment role with limited rights.

Manage upgrades

• Pin the library version in a lockfile.
• Read release notes before upgrading.
• Compare the diff of templates before and after to see unintended changes.

Keep secrets out of code

• Reference secrets by name from a secret store; do not put values in code or context files. See HashiCorp Vault for a manager that does the same job outside AWS.

Know when to drop to L1

If a property is missing from a higher level construct, use escape hatches to set it on the underlying L1 resource. Add a comment explaining why.

Cross-stack references

When one stack needs a value from another, pass the construct itself as a property. The CDK creates the export and import for you.

class StorageStack(Stack):
    def __init__(self, scope, id, **kwargs):
        super().__init__(scope, id, **kwargs)
        self.curated = s3.Bucket(self, "Curated", versioned=True)

class ReaderStack(Stack):
    def __init__(self, scope, id, curated: s3.IBucket, **kwargs):
        super().__init__(scope, id, **kwargs)
        role = iam.Role(self, "Reader",
            assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"))
        curated.grant_read(role)

storage = StorageStack(app, "Storage")
ReaderStack(app, "Reader", curated=storage.curated)

Be careful: an exported value cannot be changed while another stack uses it. Plan the stack boundaries early.

Writing your own construct

A custom construct packages a pattern your team repeats. For example, a "governed table bucket" can always apply encryption, versioning, logging and a standard lifecycle rule.

class GovernedBucket(Construct):
    def __init__(self, scope, id, *, retention_days=365):
        super().__init__(scope, id)
        self.bucket = s3.Bucket(
            self, "Bucket",
            versioned=True,
            encryption=s3.BucketEncryption.S3_MANAGED,
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            enforce_ssl=True,
            removal_policy=RemovalPolicy.RETAIN,
            lifecycle_rules=[s3.LifecycleRule(
                expiration=Duration.days(retention_days))],
        )

Now every team gets safe defaults by using one class. When the standard changes, you update the construct and release a new version.

Aspects for guardrails

An aspect visits every construct in the tree. You can use it to check a rule, or to add tags everywhere.

import jsii
from aws_cdk import IAspect, Annotations, aws_s3 as s3

@jsii.implements(IAspect)
class RequireVersioning:
    def visit(self, node):
        if isinstance(node, s3.CfnBucket):
            if node.versioning_configuration is None:
                Annotations.of(node).add_error("Bucket must enable versioning")

Aspects.of(app).add(RequireVersioning())

Suggested review checklist for a CDK pull request

Common mistakes

• Treating it as a runtime script. The code runs at synth time; logic that needs deploy time values must use tokens.
• Putting data stores and functions in one stack. A failed function change can then threaten the data store update.
• Hard coding names. Replacement of such resources fails because the name already exists.
• Renaming constructs casually. It can replace databases.
• Skipping bootstrap. Deploys fail with confusing errors.
• Not reading the diff. Permission widening goes unnoticed.
• Leaving removal policy at default for important data. Some resources are deleted with the stack.
• Overusing clever code. Deep inheritance and dynamic loops make stacks hard to follow.
• No tests. Small mistakes reach production.
• Mixing many accounts in one pipeline without guardrails. A bad change spreads widely.
• Ignoring stack limits. Large stacks hit hard caps and need splitting.
• Hand editing deployed resources. The next deploy reverts or conflicts.

Frequently asked questions

What is AWS CDK?

AWS CDK is a framework that lets you define AWS infrastructure using programming languages like TypeScript and Python. It synthesises your code into CloudFormation templates, which AWS then uses to create and update resources in a controlled way with rollback on failure.

Is the CDK the same as CloudFormation?

No. CloudFormation is the AWS service that deploys templates. The CDK is a layer above it that generates those templates from code. You get the reliability of CloudFormation with the convenience of a programming language and reusable constructs.

What is a construct?

A construct is a building block that represents one or more cloud resources. Constructs come in three levels: direct wrappers of CloudFormation resources, curated constructs with defaults and helpers, and patterns that assemble several resources for a common solution. You can compose them into your own reusable constructs.

How is AWS CDK different from Terraform?

The CDK uses general purpose languages and targets AWS through CloudFormation. Terraform uses its own declarative language, supports many providers and keeps its own state. The CDK feels natural to developers who like code abstractions, while Terraform suits teams that manage many platforms with one workflow.

What does cdk bootstrap do?

It prepares an account and region for CDK deployments by creating supporting resources such as a bucket for assets and roles used during deployment. You run it once per environment before the first deploy, and again when a new bootstrap version is required.

Can the CDK be tested?

Yes. You can write unit tests that synthesise a stack and assert that the resulting template contains the resources and properties you expect. Snapshot tests catch unintended changes, and cdk diff in a pipeline shows what a pull request will alter.

How do I stop the CDK from deleting my data?

Set a retain removal policy on stateful resources such as buckets and databases, enable versioning and backups, and consider termination protection on the stack. Also keep stable construct IDs so a rename does not cause replacement.

Does the CDK help with machine learning platforms?

It can define the storage, compute, networking, permissions, schedules and endpoints that such platforms need. It does not train or evaluate models, but it makes the surrounding environment reproducible, including short lived environments for comparing model versions.

What languages can I use with the CDK?

TypeScript is the native language, and Python, Java, C# and Go are supported through generated bindings. The concepts and constructs are the same in each, so you can choose the language your team already uses.

When should I not use the CDK?

If your infrastructure spans many clouds and services, or your team has no wish to write code for infrastructure, a declarative multi provider tool may fit better. It is also not the right layer for managing the schema or quality of the data inside tables.

Glossary

• App: The root object of a CDK program.
• Aspect: A rule applied across many constructs during synthesis.
• Asset: A file or image uploaded during deployment.
• Bootstrap: One time environment setup for CDK deployments.
• Cloud assembly: The output of synthesis.
• CloudFormation: The AWS service that deploys templates.
• Construct: A reusable building block.
• Context: Cached lookup values used in synthesis.
• Diff: A comparison of code output with the deployed stack.
• Escape hatch: Access to lower level properties of a construct.
• Grant: A helper that adds least privilege permissions.
• Logical ID: The identity of a resource inside a template.
• Removal policy: Rule for keeping or deleting a resource with its stack.
• Stack: A unit of deployment.
• Synthesis: Generating templates from code.
• Token: A placeholder resolved at deploy time.

Summary

AWS CDK lets you write AWS infrastructure in familiar programming languages and turns it into CloudFormation for safe, managed deployment. Constructs at three levels give convenience and control, grants simplify permissions, and diff, tests and pipelines add review. For tabular foundation models it supplies the AWS platform around them: intake and validation of raw tables, catalogs, training and serving compute, batch jobs for imputation, anomaly detection and classification, governance rules, and repeatable environments for comparing model versions. Protect stateful resources, keep IDs stable, test your stacks, review diffs, and treat the code as production software.

Related

It belongs to DataOps in my stack, under Data engineering and lakehouse. Others in the same group: