Infrastructure as code

Overview

Infrastructure as code (IaC) is the practice of describing servers, networks, storage, permissions and services in text files instead of clicking through consoles. The files live in version control. They are reviewed like software, tested, and applied by automation. The result is that an environment can be rebuilt on demand, compared with another one, and changed in small, safe steps.

For data and machine learning teams, IaC matters because tabular pipelines depend on many moving parts: object storage, compute clusters, schedulers, catalogs, secrets and GPUs. When these are defined as code, a training run or a feature pipeline behaves the same in development, staging and production. This article explains the idea, how it works, where it fails, and how it supports systems that feed and serve tabular foundation models. See also Terraform, Kubernetes and GitOps with Argo CD.

What it is

Infrastructure as code treats infrastructure the way a developer treats an application. Instead of a person logging into a console and creating a database by hand, an engineer writes a file that says "there should be a database with these settings". A tool reads the file and makes the real world match it.

Think of a recipe versus a meal. A meal made once from memory is hard to repeat. A written recipe can be followed by anyone, improved over time, and checked for mistakes. IaC is the written recipe for your servers.

The core ideas

• Definitions are text. Resources are described in configuration languages or general-purpose programming languages.
• Definitions are versioned. Every change has an author, a reason and a history. Rolling back means going back to an earlier version.
• Definitions are applied by tools. A program compares what you asked for with what exists and makes the needed changes.
• Definitions are tested. Syntax checks, policy checks and trial deployments catch mistakes before production.
• Definitions are reviewed. Another person reads the change before it runs.

What counts as infrastructure here

Two meanings of "code"

Some tools use a domain-specific language made for describing resources. Others let you use a general language such as Python or TypeScript. The first group is easier to read and limits what you can do. The second group allows loops, functions and tests but can become hard to follow. Both count as IaC. The key is that the definition is text kept under version control.

Where it came from

Early system administrators set up machines by hand and kept notes. As web companies grew, they needed hundreds and then thousands of servers. Hand setup did not scale and produced machines that differed in small, hidden ways.

The first response was shell scripts. Scripts helped, but they described steps rather than outcomes, and running them twice could break things. In the 2000s, configuration management tools appeared. They let administrators describe the desired state of a machine, such as "this package is installed and this service is running". Well-known examples include CFEngine, Puppet, Chef and Ansible.

Cloud computing changed the problem. When servers became API calls, you could create a whole network, not just configure a machine. Cloud providers released their own template services, and independent tools such as Terraform arrived to describe resources across many providers with one workflow.

Containers and orchestration brought another shift. With Docker and Kubernetes, the application and its runtime are packaged and declared as files. Tools such as Helm bundle these declarations. Later, the GitOps idea made the repository the single source of truth, with an agent such as Argo CD continuously pulling changes into the cluster.

Today IaC is part of the wider DevOps and platform engineering approach. It also extends into policy as code, where rules about security and cost are written as tests that run before changes are applied.

How it works

The details vary by tool, but most IaC systems follow the same loop.

Step 1: Write the desired state

You write files that describe what should exist. For example, a bucket for raw tables, a network, and a role that lets a job read the bucket.

# Example in a declarative configuration language
resource "storage_bucket" "raw_tables" {
  name       = "example-raw-tables"
  versioning = true
  encrypted  = true
}

resource "compute_cluster" "training" {
  name        = "example-training"
  node_count  = 4
  node_type   = "gpu-small"
  network_id  = network.main.id
}

The names above are placeholders. The point is the shape: a resource type, a name and a list of settings. References such as network.main.id tell the tool that the cluster depends on the network.

Step 2: Build a dependency graph

The tool reads all files and finds which resources depend on which. A cluster needs a network. A role needs a bucket. The tool builds a graph and decides an order. Independent resources can be created in parallel.

Analogy: building a house. You pour the foundation before the walls, but the electrician and plumber can work at the same time once the walls stand.

Step 3: Read the current state

The tool needs to know what already exists. It does this in one of two ways.

• It keeps a state record, a file or database listing every resource it manages and its last known settings.
• It queries the live system directly each time.

State records are fast and let the tool track resources that have no natural name. They also create a new thing to protect, because a lost or corrupted state record confuses the tool.

Step 4: Compute a plan

The tool compares desired state with current state and prepares a plan: create these, change those, destroy the others. Good tools show this plan before doing anything. Reviewers read the plan the way they read a code diff.

Step 5: Apply the plan

After approval, the tool calls the provider APIs in dependency order. If a call fails, a well-designed tool stops and records what completed so you can fix the problem and run again.

Step 6: Detect and handle drift

Drift happens when the real system changes outside the code. Someone edits a firewall in the console during an incident. The tool finds the difference on the next plan and proposes to reverse it, or you update the code to match. Regular drift checks keep the code honest.

Declarative versus imperative

Most modern IaC is declarative. The tool works out the steps. This is like telling a taxi the address instead of giving every turn.

Idempotency

An operation is idempotent if running it twice gives the same result as running it once. Declarative IaC aims for this. If the bucket already exists with the right settings, the second run changes nothing. Idempotency is what makes automation safe to retry.

Mutable and immutable infrastructure

• Mutable: you change servers in place, for example by upgrading a package on a running machine. Over time, servers differ.
• Immutable: you never edit a running server. You build a new image, launch new servers from it, and remove the old ones. Every server of a type is identical.

Immutable patterns reduce drift and make rollback simple, because the old image still exists.

Modules and reuse

Teams package common patterns into modules or templates. A "private data network" module can be used in many projects with different inputs. Modules cut duplication and spread good practice. They also need versioning, so a change to a module does not surprise every consumer at once.

Environments

A single set of definitions can produce several environments through variables. Development might use one small node, production might use ten large ones. The structure stays identical, so tests in development say something real about production.

Key concepts and terms

A worked example

Suppose a team wants an isolated environment for a tabular model pipeline: a storage bucket for input tables, a small compute pool, and a role with limited access. They want identical copies for staging and production.

The definitions

variable "env" {
  type = string
}

variable "node_count" {
  type = number
}

resource "storage_bucket" "tables" {
  name       = "example-${var.env}-tables"
  versioning = true
  encrypted  = true
}

resource "iam_role" "pipeline" {
  name = "example-${var.env}-pipeline"
  allow = [
    { action = "read",  resource = storage_bucket.tables.arn },
    { action = "write", resource = "${storage_bucket.tables.arn}/output/*" }
  ]
}

resource "compute_pool" "workers" {
  name       = "example-${var.env}-workers"
  node_count = var.node_count
  role       = iam_role.pipeline.name
}

The variable files

# staging.vars
env: staging
node_count: 1

# production.vars
env: production
node_count: 6

The workflow

1. An engineer opens a change request with the new files.
2. Automation runs a format check and a syntax check.
3. A policy check confirms that the bucket is encrypted and that no role has wildcard access.
4. The tool prints a plan for staging: three resources to create, none to change, none to destroy.
5. A reviewer reads the plan and approves.
6. The change merges, and automation applies it to staging.
7. After tests pass, the same code is applied to production with the production variable file.

A numeric walk-through of a change

Now the team needs to grow production from 6 to 8 workers. They edit one number.

The plan shows one update, zero creations and zero destructions. A reviewer sees that the change is small and safe. If the plan had shown "destroy and recreate" for the bucket, the reviewer would stop and ask why, because destroying a bucket may delete data.

Testing the definitions

Teams test IaC at several levels.

• Static checks: formatting, syntax and linting.
• Policy checks: rules like "no public buckets" run on the plan.
• Plan tests: assert that the plan contains what you expect.
• Integration tests: create a short-lived real environment, check it works, destroy it.

A small Python test for the policy layer might look like this:

def check_plan(plan):
    errors = []
    for res in plan["resources"]:
        if res["type"] == "storage_bucket":
            if not res["settings"].get("encrypted"):
                errors.append(f"{res['name']}: encryption is off")
            if res["settings"].get("public"):
                errors.append(f"{res['name']}: bucket is public")
    return errors

plan = {"resources": [
    {"type": "storage_bucket", "name": "tables", "settings": {"encrypted": True}}
]}
assert check_plan(plan) == []

The code is a toy, but real policy engines work the same way: they read the planned changes and reject the dangerous ones.

Strengths and limitations

Strengths

• Repeatability. The same definitions create the same environment every time.
• Speed. A full environment can be created in minutes.
• Review and audit. Every change has a record, an author and a reviewer.
• Safer change. The plan step shows the effect before it happens.
• Disaster recovery. A lost region can be rebuilt from the repository, provided the data has backups.
• Documentation by construction. The code is the up-to-date description of the system.
• Cost control. Temporary environments can be created and destroyed on schedule.
• Consistency. Staging truly resembles production.

Limitations

• Learning curve. Teams must learn the tool, the provider and the workflow.
• State handling. Lost, locked or corrupted state is painful. Shared state needs locking and backups.
• Drift. Emergency manual fixes break the model unless they are brought back into code.
• Secrets. Credentials must be referenced from a secret manager, never typed into definitions. See HashiCorp Vault.
• Abstraction cost. Heavy modules hide details and make debugging harder.
• Provider lag. New cloud features may take time to appear in the tool.
• Data is not code. IaC creates the database but does not move or protect the data inside it. Backups and migration are separate work.
• Destructive changes. A small edit can force a resource to be replaced, which may delete data.
• Large blast radius. One shared configuration for everything means one mistake can affect everything. Splitting into smaller units reduces risk.

How it compares with alternatives

These are not exclusive. A common pattern uses a multi-cloud tool for networks and clusters, Helm charts for applications, and a GitOps controller to keep the cluster in sync. See Terraform, Helm and Argo CD.

Role in tabular foundation models

Infrastructure as code does not train or run a model. Its role is to create and govern the environment around the model. Tabular foundation models learn from raw tables, so the way tables arrive, are stored and are served matters. This section explains the link concretely.

Reproducible pre-training environments

Pre-training a model on many tables, whether synthetic or real, needs a large and consistent cluster. Training jobs depend on specific GPU types, driver versions, storage throughput and network settings. If a job runs differently on two clusters, results become hard to compare. IaC pins these settings in code, so an experiment can be repeated and its infrastructure inspected. This supports the research practices described on /model/research.

Storage for rows and columns

Tabular data comes as files in object storage, tables in warehouses and streams from queues. IaC defines the buckets, retention rules, encryption and access roles. It can also define separate zones for raw, cleaned and curated tables, in the style of medallion architecture. When a model expects mixed types, missing values and no reliable column names, the pipeline before the model must treat each zone consistently. Code makes those zones reproducible.

Serving and in-context learning

A tabular foundation model can use in-context learning: it receives a table of labeled example rows plus new rows and predicts directly, without a training step per task. Serving this needs low-latency endpoints, GPU nodes and autoscaling. These are defined in code using Kubernetes and container images from Docker, often bundled with Helm. The context size, memory limits and scaling rules are infrastructure settings that directly affect which tables the model can handle. The design of the model itself is covered in /model/architecture.

Continual learning and safe change

When a model is updated with new data, a rollout needs controls: a canary deployment, a rollback path and a record of which version served which prediction. IaC and GitOps give that record. A model version is just another declared resource, and changing it is a reviewed commit.

Imputation, anomaly detection and classification jobs

Jobs that fill in missing values, flag unusual rows or assign semantic categories to columns are often scheduled batch work. Their schedulers, queues and worker pools can be declared in code, alongside tools such as Apache Airflow. That ensures a nightly imputation job has the same memory and permissions every night. Related ideas appear in anomaly detection.

Governance of sensitive tables

Raw tables often hold personal or confidential fields. IaC lets teams express encryption, network isolation and least-privilege roles as reviewable code, and policy as code can block a change that would expose a bucket. Secrets for database connections come from a secret manager such as HashiCorp Vault, referenced rather than embedded.

Benchmarks need stable environments

When results are compared across models, the hardware and software setup must be stated and repeatable. Declaring the environment in code is the cleanest way to publish it. See /model/benchmarks for how comparisons are framed.

What IaC does not do

It does not clean the data, choose a model, or check that a column means what you think. It only makes sure the platform under those activities is predictable. Data quality and meaning belong to catalogs, lineage and governance, such as data lineage and data catalogs.

Practical guidance and best practices

Start small

Begin with one low-risk piece, such as a storage bucket or a test network. Learn the workflow before moving critical systems under management. Importing existing resources into code is possible, but do it carefully and check the plan shows no changes afterward.

Keep everything in version control

Include definitions, variable files (without secrets), module versions and documentation. Use short-lived branches, pull requests and required reviews.

Protect the state

• Store shared state in a remote backend with locking.
• Turn on versioning and backups for the state storage.
• Restrict who can read it, because it can contain sensitive values.
• Never edit it by hand unless you have a recovery plan.

Keep secrets out of code

Reference secrets by name from a manager. Do not commit tokens, keys or passwords, even in private repositories. Rotate any credential that was committed by mistake.

Split into small units

Separate networks, shared services and applications into different configurations. A mistake in an application change then cannot delete the network. Smaller units also plan and apply faster.

Use modules wisely

Create a module when a pattern repeats three or more times. Version modules and pin consumers to a version. Avoid modules that wrap a single resource without adding value.

Review the plan, not only the code

Code can look harmless and still produce a destructive plan. Make the plan output visible in the review, and highlight replacements and deletions.

Add policy checks

Write rules for encryption, public access, tagging and instance sizes. Run them on every plan. Treat failures as blockers.

Tag everything

Add tags for owner, environment, project and cost center. Tags make cost reports and cleanup possible.

Plan for drift

Run a scheduled plan that only reports differences. Alert on unexpected changes. Decide a process for emergency manual edits: make the fix, then bring the code in line within a short deadline.

Protect data-bearing resources

Use lifecycle protections so databases and buckets cannot be destroyed by accident. Keep backups outside the same account or region when possible.

Automate through a pipeline

Run checks and plans in CI, and apply from a controlled runner with limited credentials. Pipelines such as GitHub Actions or GitLab CI/CD can host this flow. Humans should not apply from laptops in production.

Test in an ephemeral environment

For modules that matter, create a temporary copy, run checks, and destroy it. Budget limits and automatic cleanup prevent forgotten resources.

Document the entry points

A short readme per configuration should state what it manages, how to plan and apply, who owns it and how to recover.

Organize the repository

A clear layout helps new people find things. One common pattern is shown below.

infra/
  modules/
    private-network/
    object-storage/
    gpu-pool/
  environments/
    staging/
    production/
  policies/
  README.md

Modules hold reusable pieces. Each environment folder wires modules together with its own variables. The policies folder holds the checks that run in the pipeline.

Manage change over time

• Keep changes small, so a failed apply is easy to understand.
• Apply to staging first and wait for checks to pass.
• Record the reason for each change in the pull request description.
• Retire unused resources promptly, because idle GPUs and forgotten volumes cost money.
• Review module versions on a schedule, so upgrades do not pile up.

Measure the practice

Useful signals include how long it takes to build a new environment, how often drift is found, how many changes fail in apply, and how long recovery takes in a drill. These numbers show whether the practice is improving.

Common mistakes

• Mixing manual and coded changes. Console edits make the code lie. Pick one path.
• Committing secrets. Even a deleted commit remains in history.
• One giant configuration. Slow plans and large blast radius.
• Ignoring plan output. Approving changes without reading them defeats the purpose.
• No state locking. Two people applying at once can corrupt state.
• Unpinned versions. An automatic upgrade of a provider or module changes behavior silently.
• Hard-coded environment values. Copying files and editing names by hand leads to differences. Use variables.
• Destroying data by accident. A renamed resource may be seen as delete-and-create. Check plans for replacements.
• No rollback plan. Know how to return to the previous version before you apply.
• Over-abstraction. Modules with dozens of switches are harder to read than the raw resources.
• Treating IaC as a backup. It rebuilds structure, not data.
• Skipping tags and ownership. Unowned resources become costly orphans.
• Applying from personal machines. Credentials spread and history is lost.
• Never testing recovery. A rebuild process that has not been tried may not work when needed.

Frequently asked questions

What is infrastructure as code in simple terms?

It means writing down your servers, networks and services in text files and letting a tool create them for you. Because the files are stored in version control, you can review changes, repeat the setup anywhere and go back to an earlier version when something goes wrong.

Is infrastructure as code the same as configuration management?

They overlap but differ in focus. Configuration management mainly sets up the software and settings inside existing machines, while infrastructure as code mainly creates and wires the resources themselves, such as networks, databases and clusters. Many teams use both together.

What is configuration drift and why does it matter?

Drift is the gap that appears when the real system is changed by hand and no longer matches the code. It matters because the code then stops being a trustworthy description, and a later apply might undo an emergency fix or fail in surprising ways. Regular plan runs reveal drift early.

Should I use a declarative or imperative approach?

For most infrastructure, declarative is the better default because it is easier to review and safe to run repeatedly. Imperative steps still have a place for one-time migrations or glue tasks, but keep them small and wrap them so they can be retried.

How do I handle secrets in infrastructure code?

Never write them into the definitions. Store them in a dedicated secret manager and reference them by name, so the code contains only the pointer. Also restrict access to the state storage, since it can hold sensitive values, and rotate anything that was exposed.

What is a state file and do I always need one?

A state file is the tool's record of which resources it manages and their last known settings. Many tools use one for speed and to track resources accurately. Some tools query the live system instead, which avoids state storage but can be slower. If your tool uses state, keep it remote, locked and backed up.

Does infrastructure as code help machine learning projects?

Yes. Training and serving depend on consistent GPU clusters, storage, permissions and schedulers, and code makes those repeatable and auditable. It also supports safe rollouts of new model versions and lets teams publish the exact environment behind a result.

Can infrastructure as code back up my data?

No. It can recreate the database, bucket or cluster, but not the contents. You still need separate backups, snapshots and tested restore procedures for the data itself.

How do I start adopting it in an existing environment?

Pick a small, low-risk area, define it in code, and import the existing resources until the plan shows no changes. Then add reviews and a pipeline, and expand step by step. Moving everything at once usually causes confusion and outages.

What is the difference between IaC and GitOps?

IaC is the general practice of defining infrastructure in code. GitOps is a specific operating style where a repository is the single source of truth and an agent continuously syncs the running system to it. GitOps is one way to apply IaC, and it is most common for Kubernetes clusters.

Glossary

• Apply: the step that makes real changes to match the plan.
• Backend: the storage location for state.
• Blast radius: how much can break from one change.
• Canary: a small first rollout used to test a change.
• Declarative: describing the end state rather than the steps.
• Desired state: the configuration written in code.
• Drift: divergence between code and reality.
• GitOps: operating style that syncs systems from a repository.
• Idempotent: giving the same result when repeated.
• Immutable infrastructure: replacing rather than editing running servers.
• Imperative: describing the steps to run.
• Lifecycle rule: a setting that controls how a resource is created, replaced or protected.
• Module: a reusable group of definitions.
• Plan: a preview of changes.
• Policy as code: automated checks that enforce rules.
• Provider: a plugin that talks to a platform's API.
• Remote state: state stored in a shared, locked location.
• Resource: one managed object such as a bucket or a network.
• Rollback: returning to a previous known good version.
• State file: the tool's record of managed resources.
• Tag: a label attached to a resource for ownership and cost tracking.
• Variable: an input that changes a definition per environment.

Summary

Infrastructure as code turns servers, networks and services into reviewed, versioned, testable text. Its main gains are repeatability, safer change and a clear audit trail. Its main risks are state handling, drift, secrets and destructive changes, all of which have well-known controls: remote locked state, scheduled drift checks, secret managers, plan reviews and policy checks.

For tabular foundation models, IaC is the quiet layer underneath. It provides the reproducible clusters for pre-training, the governed storage for raw and curated tables, the serving endpoints for in-context prediction, and the safe rollout path for continual updates. It does not clean data or choose models, but it makes the platform predictable so that results can be trusted and repeated. Related reading: Terraform, Kubernetes, data lineage, and the model pages at /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to DataOps in my stack, under Data engineering and lakehouse. Others in the same group: