ControlNet

Overview

ControlNet is a method that adds spatial control to a pre-trained text-to-image diffusion model. Instead of relying only on a text prompt, you also give the model a guide image, such as an edge map, a depth map or a human pose skeleton. The model then generates a new picture that follows that structure while still obeying the prompt.

The idea is to attach a trainable copy of the model's encoder to a frozen original, joined by layers that start at zero. This lets the system learn a new type of condition without damaging what the base model already knows. ControlNet became a standard add-on for Stable Diffusion, and it is a good example of how to add a new input type to a large model safely. That lesson is relevant to tabular foundation models, where new kinds of conditions, such as constraints on columns or rows, must be added without retraining everything.

What it is

ControlNet is a neural network architecture and training recipe for conditional generation. It takes a pre-trained diffusion model, locks its weights, and adds a trainable branch that reads an extra condition image. The branch feeds its features back into the locked model at several depths.

• The base model is the "locked copy". It keeps its original skills.
• The control branch is the "trainable copy". It learns to read the new condition.
• Zero-initialised convolution layers join them, so training starts from the unchanged base behaviour.

Everyday analogy

Think of a skilled painter who works from a text brief alone. You now place a tracing sheet with a rough outline in front of them. The painter still uses their own style and knowledge, but they follow the outline. ControlNet is the tracing sheet and the helper who teaches the painter how to read it.

Common condition types

What it can do

• Keep a layout fixed while changing the style or content.
• Turn a sketch into a finished image.
• Reuse a pose across different characters.
• Combine several controls at once with chosen weights.
• Keep consistent structure across many images.

What it is not

• It is not a standalone image generator. It needs a base diffusion model.
• It does not replace the text prompt. It works together with it.
• It does not guarantee pixel-exact copies of the guide. It follows structure with some freedom.

Where it came from

ControlNet was introduced in 2023 by researchers working on adding conditional control to text-to-image diffusion models. It appeared at a time when open diffusion models were popular but users lacked precise control over layout.

The problem it addressed

Text prompts are good for content and style but poor at layout. Saying "a person with the left arm raised" is unreliable. Researchers had tried to fine-tune the full model on paired data, such as image and edge map, but this was costly and risked forgetting. Small datasets could lead to overfitting and a loss of image quality.

Earlier approaches

• Fine-tuning the whole model on conditional data.
• Training small adapters that inject features into the model.
• Inpainting and image-to-image, which keep part of the input but give limited structural control.
• Conditional GANs for image-to-image translation, which map edges to photos but are narrow and less flexible.

What ControlNet changed

ControlNet showed that a trainable copy of the encoder blocks, linked with zero-initialised convolutions, could learn many kinds of controls from relatively small datasets, without hurting the base model. It also showed that the approach is robust across condition types and training set sizes. Later work lightened the design, added more conditions and produced versions for newer base models.

Related work

Other methods inject control with lightweight adapters or by changing attention. They aim for lower memory use or broader conditions. ControlNet remains a common reference point and is widely supported in open tooling.

How it works

The method has a few simple parts. Each is explained below with an analogy.

Part 1: the locked copy

The base diffusion model, typically a U-Net, has an encoder half that shrinks features, a middle block, and a decoder half that grows them back. The weights of this model stay frozen. Because they never change, the base model's knowledge is safe. Think of it as a master craftsperson whose skills you do not want to disturb.

Part 2: the trainable copy

ControlNet makes a copy of the encoder blocks and the middle block. This copy is trainable. It receives the usual noisy latent and the text embedding, and also a condition signal. The condition image is first processed by a small convolutional network to bring it to the same size as the latent. Think of the copy as an apprentice who watches the master but also reads the tracing sheet.

Part 3: zero convolutions

The outputs of the trainable copy are joined to the locked model through convolutions whose weights and biases start at zero. At the start of training, these layers output zero, so the combined model behaves exactly like the base model. The zero layers then learn, little by little, how much of the control signal to inject.

Why does this matter? Without it, the random outputs of a new branch could add noise to the base model's features at the start and harm it. With zero layers, the first training step does no harm. The gradients still flow, because the weights get non-zero gradients even when they start at zero. The signal grows only when it helps.

Part 4: injection points

The control features are added to the skip connections and the middle block of the locked U-Net. This means the condition influences the decoder at several scales, from coarse layout to fine edges. Like giving a painter guidance at the sketch, the underpainting and the final detail stages.

Training in steps

1. Prepare pairs: a target image, a caption and a condition image derived from the target.
2. Derive the condition by running an edge detector, a depth estimator or a pose detector on the target.
3. Encode the target to a latent and add noise at a random step.
4. Feed the noisy latent, the step and the caption to the locked model.
5. Feed the same inputs plus the condition to the trainable copy.
6. Add the copy's zero-convolved outputs into the locked model's features.
7. Compare the predicted noise with the true noise and update only the ControlNet.

An optional trick is to drop the caption at random during training. This forces the control branch to understand the condition even without text, which can improve the response to the guide image.

Sampling in steps

1. Prepare the guide image, such as an edge map from a reference photo.
2. Write a prompt.
3. Start from random noise in latent space.
4. At each step, run the locked model with the control features added.
5. Use guidance and a scheduler as usual.
6. Decode the final latent to an image.

Control strength

A control weight scales the injected features. A weight of one gives the full effect. A lower weight loosens the guide, which allows more creative freedom. A higher weight can make the image stick too closely to the guide and reduce quality. Two other settings help: when to start applying control and when to stop. Applying control only in the early steps fixes layout and lets later steps add natural detail.

Combining controls

Several ControlNets can run at once. For example, pose plus depth, or edges plus segmentation. Each has its own weight. Combining too many can cause conflicts, in the way two people giving contradictory directions can confuse a painter. Start with one control and add others only if needed.

Preprocessors

The guide image usually comes from a preprocessor.

• An edge detector for edge-based control.
• A monocular depth estimator for depth control.
• A pose estimator for pose control.
• A segmentation model for region control.

The quality of the preprocessor sets the quality of the guide. A noisy depth map produces a messy result.

Efficiency

Training a ControlNet needs less data than training a base model. Because the base model is frozen, there is no need to store optimizer state for it. Memory use is still higher than for a small adapter, since the copy of the encoder is large. Lighter variants cut the size of the copy.

Why copy the encoder and not the whole model

The encoder half of the U-Net is where the model builds up an understanding of the input at several scales. A condition such as an edge map is naturally read at those scales. The decoder half then uses skip connections to turn that understanding back into pixels. By copying only the encoder and the middle block, ControlNet places the new reading ability exactly where it can influence the decoder, without duplicating the part that paints the final image.

Why not just add the condition as another input channel

Concatenating the condition to the input is simple, but it forces the base model to relearn its first layers, which risks forgetting. It also gives the condition only one entry point at the start. ControlNet gives the condition many entry points, which is better for fine structure.

Understanding the "sudden convergence" effect

During training, the model often seems to ignore the condition for a long time and then suddenly starts to follow it. This happens because the zero layers grow slowly at first and the base model can already make a good image without the guide, so the loss gives little push. Once the zero layers are large enough, the control takes effect quickly. So a long flat stretch is normal, and stopping early is a common error.

How guidance interacts with control

Classifier-free guidance uses a conditional and an unconditional prediction. With a ControlNet, you decide whether the control applies to both, or only to the conditional one. Applying it to both keeps the structure stable at any guidance scale. Applying it only to the conditional one makes the structure depend on the guidance scale. Tools differ in their defaults, so check what yours does.

Typical control types and when to pick them

• Edges when you want to keep the outline of an object and change everything inside it.
• Depth when you want a new scene with the same spatial arrangement, ignoring texture.
• Pose when you want a person in a given posture, with any clothes or setting.
• Segmentation when you want regions of sky, building and road placed exactly.
• Scribble when the input is a rough drawing with missing detail.
• Line art when the input is a clean drawing to be coloured or rendered.

Key concepts and terms

A worked example

The example uses the Hugging Face Diffusers library to generate an image that follows the edges of a reference photo. Model names are placeholders.

import cv2
import numpy as np
import torch
from PIL import Image
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline

# 1) Make a Canny edge map from a reference image
ref = np.array(Image.open("example_reference.png").convert("RGB"))
edges = cv2.Canny(ref, 100, 200)
edges = np.stack([edges] * 3, axis=-1)
control_image = Image.fromarray(edges)

# 2) Load the control branch and the base model
controlnet = ControlNetModel.from_pretrained(
    "example-org/controlnet-canny", torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "example-org/stable-diffusion-checkpoint",
    controlnet=controlnet,
    torch_dtype=torch.float16,
).to("cuda")

# 3) Generate with a prompt and the guide
generator = torch.Generator(device="cuda").manual_seed(7)
image = pipe(
    prompt="a modern office lobby, warm light, photorealistic",
    image=control_image,
    num_inference_steps=25,
    guidance_scale=7.0,
    controlnet_conditioning_scale=0.9,
    control_guidance_start=0.0,
    control_guidance_end=0.8,
    generator=generator,
).images[0]
image.save("example_controlled.png")

Reading the example

• The Canny step makes the guide. The two thresholds decide how many edges are kept.
• The control branch is loaded separately and attached to the base pipeline.
• The conditioning scale of 0.9 keeps strong but not total control.
• Ending control at 80 percent of the steps lets the last steps add natural detail.

Two controls at once

controlnets = [
    ControlNetModel.from_pretrained("example-org/controlnet-depth", torch_dtype=torch.float16),
    ControlNetModel.from_pretrained("example-org/controlnet-pose", torch_dtype=torch.float16),
]
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "example-org/stable-diffusion-checkpoint",
    controlnet=controlnets,
    torch_dtype=torch.float16,
).to("cuda")
image = pipe(
    prompt="two dancers on a stage",
    image=[depth_image, pose_image],
    controlnet_conditioning_scale=[0.6, 0.8],
).images[0]

A numeric walk-through of zero initialisation

Let a control feature be c and a zero convolution have weight w. The injected value is w times c. At the start, w is 0, so the injected value is 0 whatever c is. The gradient with respect to w is proportional to c times the upstream gradient, which is not zero. After a small update, w becomes, say, 0.01, and the injection is 0.01 times c. As training goes on, w grows only where it helps reduce the loss. The numbers are illustrative.

Parameter count intuition

If the encoder half of the base model has about a third to a half of its parameters, a full ControlNet copy adds that many trainable parameters on top. That is much smaller than training the base model, but larger than a thin adapter. This is the reason for lighter variants that keep fewer blocks or narrower layers.

Strengths and limitations

Strengths

• Precise structural control that text alone cannot provide.
• The base model stays intact, so image quality and general skills are kept.
• Trains on modest datasets, even a few thousand pairs for simple conditions.
• Many conditions are available, and they can be mixed.
• Works with the large ecosystem around the base model.
• Control strength and range are adjustable at inference time.

Limitations

• Extra memory and compute for the control branch at inference.
• Each condition type usually needs its own trained branch.
• A ControlNet is tied to the base model family it was trained on. A branch for one version often does not work with another.
• Quality depends on the preprocessor. Bad guides give bad images.
• Too much weight makes images stiff, and too little weight ignores the guide.
• Conflicts arise when several controls disagree or when the guide conflicts with the prompt.
• It controls structure, not fine identity or exact text.

Safety and ethics

• Pose and edge control make it easier to place a chosen subject in a scene, which can be misused for misleading images.
• Label synthetic images clearly.
• Respect consent, privacy and licence terms.

How it compares with alternatives

How to choose

• Pick ControlNet when layout, pose or depth must be preserved.
• Pick a low-rank adapter when you want to teach a style or a subject.
• Pick image-to-image for a quick, loose redraw.
• Pick inpainting when only a small region changes.
• Combine them: an adapter for style, a ControlNet for layout.

Role in tabular foundation models

ControlNet works on images, but the design principle is directly useful for models that learn from raw tables. The principle is: add a new kind of condition to a large pre-trained model without damaging it.

Conditioning a table generator

A diffusion model for tabular data, like the image models above, generates rows from noise. Practical users often want rows that satisfy constraints: a column must stay within a range, two columns must stay consistent, or a class must appear in a given proportion. A ControlNet-like branch can read such a constraint as a condition and inject it into the denoiser. The base generator keeps its general knowledge of what tables look like, and the branch learns to steer.

Row and column structure

In images, a condition is a map aligned with the pixel grid. In tables, the equivalent is a condition aligned with the cell grid: a mask that marks which cells are fixed, a target for a column, or a template that fixes a column pattern. Because the branch is aligned with the base encoder blocks, the condition can affect each layer in the right place.

Mixed types and missing values

A control mask can mark observed cells and missing cells. The generator then fills the missing ones while the observed ones stay fixed. This is conditional imputation, much like inpainting. The control weight offers a dial between strict respect for observed values and freedom to correct noisy ones.

No column names

If headers are missing, a condition can still pin down structure by position or by statistics. For example, a condition might say "column 3 is a positive skewed amount" without a name. A branch trained on many tables with such descriptors can learn to follow them. This supports semantic and sector classification by providing a clean way to feed column-level hints into the model.

Pre-training on synthetic and real tables

The two-part design matches a common training plan: first pre-train a base model on very many synthetic and real tables, as in TabPFN, then train small control branches for particular tasks or domains. The base stays frozen, which protects general ability.

In-context learning

In-context learning in tabular models means the model reads example rows in its input and predicts for new rows with no weight update. A control branch is a weight-based alternative: it learns a condition once and applies it forever. The two are complementary. Use in-context examples for quick adaptation and a control branch for a stable, repeated condition.

Continual learning

The zero-initialised branch is a model of safe continual learning. A new domain gets a new branch. The old branches and the base are untouched, so nothing is forgotten. Switching branches is like switching a tool, not retraining the whole system. See /model/architecture for how such modular designs fit with table models.

Anomaly detection

A condition can describe "normal" structure. Rows that cannot be generated or reconstructed under the normal condition, or that need a large control signal to fit, are candidates for anomalies. This is a reconstruction-based view and should be checked against simpler detectors.

Cautions

• A tabular branch needs training pairs of tables and conditions, which must be built carefully.
• Hard constraints are better enforced by post-processing or by a rule check than by a soft control weight alone.
• Test for privacy leakage if the generator is trained on sensitive tables.

For measured results on tabular tasks, see /model/benchmarks. For research context, see /model/research.

Practical guidance and best practices

Getting good guides

• Use a clean, high-resolution reference.
• Tune the edge thresholds so you keep the main contours, not every texture.
• Check the depth map by eye before generating.
• Resize the guide to the output size. A mismatch causes distortion.
• Crop to the subject, since empty space wastes control capacity.

Choosing settings

Prompting with control

• Describe content and style, not layout. The guide covers layout.
• Avoid prompts that conflict with the guide, such as asking for a different pose than the one shown.
• Use a negative prompt to remove artifacts.

Training your own branch

1. Collect a few thousand images that represent your domain.
2. Create condition images with a reliable preprocessor.
3. Write simple, accurate captions.
4. Start from a trusted base model and keep it frozen.
5. Use a small learning rate and monitor validation images.
6. Look for the "sudden convergence" moment, when the model starts to follow the guide, and do not stop before it.
7. Test with prompts and guides that were not in the training set.

Deployment

• Load the base and the control branch in half precision.
• Cache preprocessed guides when the same reference is reused.
• Limit the number of simultaneous controls to control latency.
• Pin versions of the base, branch and preprocessor together.
• Consider exporting to an optimised runtime such as ONNX Runtime if latency matters.

Evaluation

• Measure how well the output follows the guide, by re-extracting edges or depth from the output and comparing to the guide.
• Measure prompt following separately.
• Review samples by hand, since automatic image scores miss structural errors.
• Test with unusual guides, such as thin lines or very busy scenes.

Common mistakes

• Using a branch trained for one base model version with a different base.
• Setting the control weight too high and getting stiff, over-sharp images.
• Feeding a guide of the wrong size or colour mode.
• Using a low-quality preprocessor and blaming the model.
• Writing a prompt that contradicts the guide.
• Stacking many controls at full weight.
• Training on too few or too uniform pairs, so the branch fits only a narrow case.
• Stopping training before the model starts following the guide.
• Forgetting to freeze the base model during training.
• Treating a soft control as a hard constraint.

Troubleshooting checklist

1. Output ignores the guide. Raise the control weight, check the guide size and check the branch matches the base.
2. Output looks burnt or rigid. Lower the weight and end control earlier.
3. Strange duplicated shapes. Check the resolution and the guide alignment.
4. Slow generation. Reduce the number of controls, steps or resolution.
5. Out of memory. Use half precision, offload or a lighter branch.

Example use cases

1. Architecture and interior design: keep the floor plan or room outline, change materials and lighting.
2. Character design: keep a pose sheet, vary outfits and settings.
3. Product imagery: keep a product outline, vary backgrounds.
4. Animation: keep consistent poses across frames, then refine each.
5. Data augmentation: keep object layout, vary appearance, to enlarge a training set with controlled variation.
6. Education: turn a rough diagram into a clean illustration.

Workflow template

1. Define the goal and the structure that must stay fixed.
2. Choose one control type that captures that structure.
3. Prepare and inspect the guide.
4. Generate a few samples with different seeds.
5. Adjust the weight, the range and the prompt.
6. Add a second control or an adapter only if a gap remains.
7. Save the settings with the seed for repeatability.

Frequently asked questions

What is ControlNet?

ControlNet is an add-on network that lets a pre-trained diffusion model follow a guide image, such as edges, depth or a pose skeleton, along with the text prompt. It keeps the original model frozen and trains a copy of its encoder that reads the guide.

What are zero convolutions?

Zero convolutions are convolution layers that start with all weights and biases at zero. They connect the trainable branch to the frozen model, so at the start of training the added signal is zero and the base behaviour is unchanged. Gradients still update them, so the control grows gradually as it helps.

Does ControlNet work with any model?

No. A ControlNet is trained against a specific base model family, and it usually works only with that family or very close fine-tunes of it. When the base changes substantially, the branch must be retrained or replaced by one made for the new base.

Can I use more than one ControlNet at once?

Yes. Several branches can be combined, each with its own weight, for example depth plus pose. Too many controls can conflict, so add them one at a time and tune the weights.

How much data is needed to train a ControlNet?

Simple conditions can be learned from a small to moderate dataset, often thousands of image pairs, because the base model already knows how to make images. Complex or unusual conditions need more data and careful pair construction.

How is ControlNet different from LoRA?

LoRA modifies model weights with low-rank updates and is mostly used to teach a style or subject. ControlNet adds a separate branch that reads a spatial condition image, so it controls layout and structure. They can be used together.

Why does the output sometimes ignore my guide?

Common causes are a control weight that is too low, a guide image that is the wrong size or noisy, a branch that does not match the base model, or a prompt that contradicts the guide. Check each in that order.

Is ControlNet only for images?

The released form works on images, but the design idea, a trainable branch joined to a frozen model by zero-initialised layers, applies to other data. It can be adapted for conditions on other kinds of generative models, including those that generate tables, although this requires new training.

Can ControlNet help with synthetic tabular data?

Only by analogy. The image version does not read tables. The pattern of a frozen generator plus a trained condition branch is a sound way to steer a table generator toward constraints, and it should be paired with rule checks and privacy tests.

Does ControlNet slow down generation?

Yes, somewhat. The control branch runs at every step alongside the base model, so each step costs more than plain generation. Lighter branches, fewer controls and ending control early all reduce the cost.

Glossary

• Adapter: A small module that adds capability to a frozen model.
• Base model: The pre-trained model that a control branch is attached to.
• Branch: The trainable part added next to the frozen model.
• Canny: A classic edge detection method.
• Conditioning: Extra input that shapes generation.
• Control weight: The strength of the control signal.
• Depth estimation: Predicting distance from a single image.
• Frozen: Not updated during training.
• Guide image: The image that provides structure, such as edges or depth.
• Inpainting: Filling in a masked part of an image.
• Latent: A compressed internal representation of an image.
• Overfitting: Fitting training data too closely and failing on new data.
• Preprocessor: A tool that converts a reference into a guide image.
• U-Net: The shrink-and-grow network used for denoising.
• Zero initialisation: Starting weights at zero so a new layer does no harm at first.

Summary

ControlNet gives a pre-trained diffusion model precise spatial control. A locked copy keeps the base model's skills, a trainable copy reads a guide image, and zero-initialised convolutions join them so training begins from unchanged behaviour. The result follows edges, depth, pose and other structures while still honouring the text prompt.

It is powerful but tied to a base model, adds compute and depends on good preprocessors. For tabular work, its lasting lesson is the safe-extension pattern: keep a strong base frozen, train a small conditioned branch, and switch branches by domain or constraint. That helps with conditional generation, imputation, semantic labelling and continual learning. Related reading includes Stable Diffusion, Diffusion models, LoRA, TabPFN and /model/architecture.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: