Terraform

Overview

Terraform is an infrastructure as code tool. You write configuration files that describe the cloud resources you want, such as networks, storage buckets, databases and clusters. Terraform compares that description with what already exists, shows you a plan of the changes, and then applies them through the provider APIs. The result is infrastructure that is reproducible, reviewable and easy to rebuild.

Teams that run data platforms and machine learning systems use Terraform to create the storage, compute, permissions and networking those systems need. A tabular data warehouse, a feature pipeline, a model serving cluster and the access rules around them can all be declared in files, kept in version control and changed through pull requests rather than by clicking in a console.

What it is

Terraform is a command line tool and a language. The language is called HCL, short for HashiCorp Configuration Language. It is declarative: you state the end result, not the steps. Terraform works out the steps.

It has five main ideas.

• Configuration. Files ending in .tf that declare resources, variables and outputs.
• Providers. Plugins that talk to a platform's API, for example a cloud, a database or a monitoring service.
• State. A record of what Terraform manages and the real identifiers of those objects.
• Plan. A preview of what will be created, changed or destroyed.
• Apply. The execution of that plan.

An everyday analogy

Imagine ordering a kitchen renovation. You give the builder a drawing of the final kitchen, not a list of hammer swings. The builder looks at the current room, notes the differences, and shows you a work list for approval. Terraform is the builder. The drawing is your configuration, the survey of the current room is the state, and the work list is the plan.

Declarative versus imperative

An imperative script says "create a bucket, then add a policy". If run twice it may fail or duplicate. A declarative file says "there is a bucket with this policy". Running Terraform twice is safe, because the second run finds nothing to change. This property is called idempotence.

Where it came from

Terraform was created by HashiCorp and first released in the mid 2010s. It followed a wave of configuration management tools such as Puppet, Chef and Ansible, which focused on configuring servers. Terraform focused on provisioning the infrastructure itself, across many providers, with a plan step that made changes visible before they happened.

Cloud providers also built their own templating systems, such as CloudFormation on AWS. Terraform's pitch was one language and workflow across many clouds and services. A large registry of community and vendor providers and modules grew around it.

In 2023 the licence of the core project changed from an open source licence to a source available one, which led to a community fork under an open governance foundation. Today there are two closely related lines of tools that share most syntax and concepts. The ideas described here apply to both. For a different approach to the same problem, see AWS CDK, which uses general purpose programming languages.

How it works

Terraform follows a cycle: write, initialise, plan, apply, and later change or destroy.

Step 1: Write configuration

You describe resources in .tf files.

terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.region
}

variable "region" {
  type    = string
  default = "us-east-1"
}

resource "aws_s3_bucket" "raw_tables" {
  bucket = "example-internal-raw-tables"
  tags = {
    purpose = "raw-tabular-data"
  }
}

output "bucket_name" {
  value = aws_s3_bucket.raw_tables.bucket
}

Step 2: Initialise

terraform init downloads the providers and sets up the backend where state is stored. Think of it as unpacking the tools before starting work.

Step 3: Build the dependency graph

Terraform reads references between resources. If a database refers to a subnet, the subnet must exist first. It builds a graph, and creates independent resources in parallel. You rarely state order by hand.

Step 4: Refresh and compare

Terraform reads the state and, by default, asks the provider for the current status of each resource. It then compares your configuration with the real world.

Step 5: Plan

terraform plan prints the differences. Each resource is marked as create, update in place, replace or destroy. You review the plan, ideally in a pull request. Saving the plan to a file lets you apply exactly what was reviewed.

Step 6: Apply

terraform apply makes the calls to the provider APIs. The state is updated as each resource finishes. If something fails midway, the state records what did succeed, and the next run continues from there.

Step 7: Change and destroy

You edit the files and repeat. terraform destroy removes everything the configuration manages. This is handy for temporary test environments and dangerous in production, so protect it with reviews and lifecycle rules.

State and why it matters

State maps your configuration names to real object identifiers. Without it, Terraform could not know that aws_s3_bucket.raw_tables is the bucket it created earlier. Treat state as sensitive: it can contain secrets in plain form. Store it in a remote backend with encryption and locking, so two people cannot apply at the same time.

Modules

A module is a folder of configuration used as a reusable unit. A "data lake" module might create a bucket, encryption, access logging and a read-only role. Teams call it with different inputs for each environment.

module "lake_dev" {
  source      = "./modules/data-lake"
  name        = "dev-tables"
  environment = "dev"
  readers     = ["analyst-role"]
}

Key concepts and terms

A worked example

This example builds a small storage and access layer for a tabular data project: a bucket for raw files, a bucket for curated tables, and a role that can read the curated one.

Files

infra/
  main.tf
  variables.tf
  outputs.tf
  envs/
    dev.tfvars
    prod.tfvars

variables.tf

variable "environment" {
  type        = string
  description = "dev or prod"
  validation {
    condition     = contains(["dev", "prod"], var.environment)
    error_message = "environment must be dev or prod."
  }
}

variable "region" {
  type    = string
  default = "us-east-1"
}

main.tf

locals {
  prefix = "example-${var.environment}"
}

resource "aws_s3_bucket" "raw" {
  bucket = "${local.prefix}-raw"
}

resource "aws_s3_bucket" "curated" {
  bucket = "${local.prefix}-curated"
}

resource "aws_s3_bucket_versioning" "curated" {
  bucket = aws_s3_bucket.curated.id
  versioning_configuration {
    status = "Enabled"
  }
}

data "aws_iam_policy_document" "read_curated" {
  statement {
    actions   = ["s3:GetObject", "s3:ListBucket"]
    resources = [
      aws_s3_bucket.curated.arn,
      "${aws_s3_bucket.curated.arn}/*",
    ]
  }
}

resource "aws_iam_policy" "read_curated" {
  name   = "${local.prefix}-read-curated"
  policy = data.aws_iam_policy_document.read_curated.json
}

Commands

terraform init
terraform plan  -var-file=envs/dev.tfvars -out=dev.plan
terraform apply dev.plan

Reading the plan

The plan lists four resources to add: two buckets, a versioning setting and a policy, plus one data source read. A line such as Plan: 4 to add, 0 to change, 0 to destroy is the summary. If later you rename a resource in the file, the plan may show one destroy and one create. That is a warning sign. Use a moved block or a state move to tell Terraform it is the same object.

moved {
  from = aws_s3_bucket.raw
  to   = aws_s3_bucket.raw_tables
}

Detecting drift

If someone edits the bucket in a console, the next terraform plan shows a change that would revert it. That is drift detection built in. In a regular schedule, a pipeline can run plan only and alert when it is not empty. See GitLab CI/CD and GitHub Actions.

Strengths and limitations

Strengths

• One workflow across many clouds and services.
• The plan step makes changes visible before they happen.
• Large ecosystem of providers and modules.
• Declarative files are readable and fit code review.
• Dependency graph and parallelism are handled for you.
• Reproducible environments: dev, staging and production from the same code.
• Easy to tear down temporary environments.

Limitations

• State is a critical, sensitive asset and must be managed with care.
• HCL is limited as a language; complex logic can become awkward.
• Provider bugs and API quirks can cause surprising plans.
• Renames and refactors can look like destroy and create.
• Drift from manual changes needs process to prevent.
• Large monolithic configurations get slow and risky. Split by system and environment.
• It describes infrastructure, not the data inside it. Schema changes of tables belong to other tools such as dbt or migration tools.
• Licence changes in the ecosystem mean you should check which distribution your organisation uses.

How it compares with alternatives

A common pattern is to use Terraform for the cloud foundation (networks, clusters, storage, identities) and a GitOps tool for workloads inside the cluster.

Role in tabular foundation models

A tabular foundation model learns from raw tables. It reads rows and columns, copes with mixed types and missing values, may see no meaningful column names, and is pre-trained on large numbers of synthetic and real tables so it can then learn a new table from a few examples through in-context learning. Terraform does not do any of that learning. Its role is the platform underneath: storing the tables, giving the model compute, controlling who can touch what, and making the whole setup repeatable.

Storage for tables and artifacts

Pre-training data, synthetic table generators' outputs, evaluation sets, model checkpoints and logs all need durable storage with the right lifecycle and encryption. Terraform declares those buckets, versioning, retention rules and access policies. A new environment for an experiment can be created in minutes with the same settings as production.

Compute for training and serving

Pre-training uses accelerator clusters. Serving uses smaller, always on endpoints. Terraform can declare node pools, instance types, autoscaling limits and networking for both. Because compute is expensive, being able to destroy a training cluster by changing one file, and recreate it later, is a practical cost control. See Kubernetes and Docker.

Governance of sensitive tables

Tables often hold personal or confidential data. Terraform expresses who can read which dataset, with least privilege roles and audit logging. Since this lives in code, auditors can review the exact access rules. Secrets should come from a manager rather than from the files. See HashiCorp Vault.

Data platform pieces that feed models

Warehouses, catalogs, streaming connectors and lineage services are all infrastructure that can be declared. For example Snowflake, Delta Lake storage layers, Kafka Connect workers, and the AWS Glue Data Catalog are commonly provisioned this way. Check the provider documentation for what a given service supports.

Supporting imputation, anomaly detection and classification

Tasks such as filling missing values, flagging anomalous rows and classifying a table's domain run as batch jobs or services. Terraform provisions the queues, schedules, containers and permissions around them. See Imputation networks and Anomaly detection.

Continual learning environments

When a model is updated over time, you want a candidate environment that mirrors production, a test run against held out tables, and a clean switch. Infrastructure as code makes the mirror cheap and exact, which makes comparisons fair.

Keeping the contract between data and model

Schema checks and contract checks run in pipelines, and the pipeline infrastructure itself can be declared in Terraform. See Schema-drift detection and Data contract validation. Context on the model side is at /model/architecture, /model/research and /model/benchmarks.

Honest boundaries

Terraform does not train, evaluate or version a model, and it does not understand tables. It supplies a stable base. Model tracking belongs to tools like MLflow.

Practical guidance and best practices

Organise code sensibly

• Split by system and environment rather than one giant configuration.
• Keep modules small, with clear inputs and outputs.
• Use consistent naming and tagging.
• Keep environment differences in variable files, not copy pasted code.

Protect state

• Use a remote backend with encryption and locking.
• Restrict who can read state, since it may contain secrets.
• Never edit state by hand unless you understand the state commands.
• Back up state through versioned storage.

Review every plan

• Run plan in pull requests and post the output for reviewers.
• Apply the saved plan that was reviewed.
• Treat any destroy line in production as needing a second look.
• Use prevent_destroy on critical resources such as primary databases.

Pin versions

• Pin the Terraform version and provider versions.
• Commit the dependency lock file.
• Upgrade deliberately and test on a non production environment first.

Handle secrets safely

• Do not put secrets in .tf files or variable files in git.
• Mark outputs and variables as sensitive.
• Prefer identity based access and short lived credentials.
• Remember that sensitive values may still appear in state.

Automate

• Run formatting and validation in CI: terraform fmt -check and terraform validate.
• Add static security scanning for common misconfigurations.
• Run scheduled plans to detect drift.
• Limit apply to a pipeline with approval, not to laptops.

Test infrastructure code

• Use terraform validate and plan against a sandbox account.
• Write tests for modules where the tooling supports it.
• Keep examples in each module so changes can be tried quickly.

Plan for rollback

Infrastructure rollbacks are not always possible. Deleted data is gone. Take backups, use versioning and design changes to be reversible, for example by creating the new resource before removing the old.

Variables, locals and outputs in practice

Inputs make a configuration reusable. Use variable blocks for anything that changes between environments, locals for derived names, and output for values that other systems need.

variable "retention_days" {
  type    = number
  default = 90
}

resource "aws_s3_bucket_lifecycle_configuration" "raw" {
  bucket = aws_s3_bucket.raw.id

  rule {
    id     = "expire-old-raw"
    status = "Enabled"
    filter {}
    expiration {
      days = var.retention_days
    }
  }
}

Creating several similar resources

Use for_each over a map when you need several similar objects, such as one bucket per data domain. Keys give each resource a stable address, so adding or removing one entry does not shift the others.

variable "domains" {
  type    = set(string)
  default = ["sales", "support", "sensors"]
}

resource "aws_s3_bucket" "domain" {
  for_each = var.domains
  bucket   = "example-${each.key}-tables"
}

A typical pipeline for infrastructure changes

Environments and promotion

Keep one codebase and one variable file per environment. Promote a change by applying it first in development, then staging, then production. Differences should be values (sizes, counts, names), never structure. If production needs a resource that development does not, add a boolean variable and a conditional count, and document why.

Importing existing resources

Many teams start with infrastructure that was built by hand. The usual path is to write the configuration, import each existing object into state, and run plan until it shows no changes. Do this in small slices, one system at a time, and commit after each slice.

Cost and tagging

Tags declared in code make cost reports meaningful. Put a small set of required tags on every resource, such as owner, environment and purpose. Use a default tags feature of the provider when available, so no resource is missed.

Common mistakes

• Storing state locally on a laptop. It gets lost or diverges.
• Committing state or secrets to git. Both are sensitive.
• Applying without reading the plan. Surprises happen in production.
• One huge configuration for everything. Slow plans and a large blast radius.
• Hand editing resources in the console. Creates drift and later reverts.
• Renaming resources without moved blocks. Triggers destroy and create.
• Leaving provider versions unpinned. Behaviour changes after an upgrade.
• Using count where for_each is clearer. Index based changes can reshuffle resources.
• Over abstracted modules. Too many options make them harder than raw resources.
• Forgetting data lifecycle. Buckets with no retention or encryption rules.
• Ignoring cost. Leaving large clusters running because nobody owns the teardown.
• Mixing application deploys and infrastructure in one change. Hard to review and roll back.

Frequently asked questions

What is Terraform used for?

Terraform is used to create and manage infrastructure such as networks, storage, databases, clusters and permissions from configuration files. It lets teams review changes, repeat environments and keep a record of what exists, instead of building things by hand in a console.

What is Terraform state?

State is a file or remote record that maps the resources in your configuration to the real objects Terraform created, along with their identifiers and attributes. Terraform uses it to compute what must change. It can contain sensitive values, so it should be stored remotely with encryption and locking.

What is the difference between terraform plan and terraform apply?

Plan shows what Terraform would do, listing resources to create, change or destroy, without changing anything. Apply carries out those changes. A good habit is to save the plan to a file during review and apply exactly that file.

What is a Terraform provider?

A provider is a plugin that lets Terraform manage resources on a particular platform, such as a cloud, a database or a monitoring service. It translates your declared resources into API calls and reports their current status back.

What is drift and how do I handle it?

Drift is when real infrastructure differs from what Terraform expects, usually because someone changed it manually. Running plan reveals it. You can either update the configuration to match the change or apply to restore the declared state. Scheduled plans and restricted console access reduce drift.

How is Terraform different from AWS CDK?

Terraform uses its own declarative language and works across many providers. AWS CDK uses general purpose programming languages and generates CloudFormation templates for AWS. CDK offers loops, types and abstractions in familiar languages, while Terraform offers a uniform workflow across clouds and services.

Can Terraform manage Kubernetes and applications?

It can create clusters and, with the right providers, some objects inside them. Many teams prefer to use Terraform for the cloud foundation and a GitOps tool for workloads inside the cluster, because application changes happen more often and have different safety needs.

How does Terraform help machine learning and data teams?

It makes storage, compute, networking and access rules for data and model workloads reproducible and reviewable. Teams can recreate a training or test environment from code, keep production and staging aligned, and show auditors exactly who can read which dataset.

Is it safe to run terraform destroy?

It is safe only when you intend it and understand the plan. It removes everything the configuration manages, which is useful for temporary environments. In production, protect critical resources with lifecycle rules, backups and approval steps so a mistaken destroy cannot remove important data.

Should secrets go in Terraform files?

No. Keep secrets out of configuration and out of version control. Fetch them from a secret manager at run time or use identity based access. Be aware that values passed through Terraform can still end up in state, so protect state accordingly.

Glossary

• Apply: The step that executes planned changes.
• Backend: The storage and locking mechanism for state.
• Blast radius: How much can be harmed by one mistaken change.
• Data source: A read only lookup of existing information.
• Declarative: Describing the desired result instead of the steps.
• Drift: Difference between real infrastructure and its declared form.
• HCL: The configuration language of Terraform.
• Idempotent: Safe to run repeatedly with the same result.
• Import: Adopting an existing object into state.
• Lock file: A file that pins exact provider versions.
• Module: A reusable folder of configuration.
• Output: A value published by a configuration or module.
• Plan: A preview of intended changes.
• Provider: A plugin for a platform's API.
• Resource: A declared infrastructure object.
• State: The record of managed objects.
• Workspace: A named state instance for one configuration.

Summary

Terraform lets you describe infrastructure in files, preview changes with a plan, and apply them through provider APIs, using state to remember what it manages. It brings review, repeatability and drift detection to cloud resources. For tabular foundation models it does not learn from tables, but it provisions the storage, compute, networking and access control that hold pre-training data, checkpoints, evaluation sets and serving endpoints, and it makes test and candidate environments cheap to recreate. Protect state, review every plan, pin versions, keep secrets out of files, split large configurations, and use it alongside pipelines and GitOps tools for safe, reproducible platforms.

Related

It belongs to DataOps in my stack, under Data engineering and lakehouse. Others in the same group: