Stable Diffusion

Overview

Stable Diffusion is an open text-to-image model. You give it a text prompt and it produces an image that matches the description. It belongs to the family of latent diffusion models, which learn to remove noise step by step, and which do the work in a compressed "latent" space instead of directly on pixels.

Working in latent space is the key idea. It makes the model much cheaper to train and run than pixel-space diffusion, so it fits on a single consumer GPU. The open release of the weights also created a huge ecosystem of tools, fine-tunes and add-ons such as ControlNet and LoRA. Beyond art, the model is used for synthetic data, design, and research on generative modelling. Its core mechanism, learning data by learning to denoise it, is also used in generative models for tables, which is why it appears in the context of tabular foundation models.

What it is

Stable Diffusion is a latent diffusion model for image generation. It is made of three main networks that work together.

• A variational autoencoder, or VAE, which compresses images into a small latent grid and expands them back.
• A U-Net, which learns to predict and remove noise in the latent grid.
• A text encoder, usually a CLIP-style model, which turns the prompt into vectors that guide the U-Net.

Around these sits a scheduler, which decides how noise is added and removed over time, and a guidance method that controls how strongly the image follows the prompt.

Everyday analogy

Imagine a sculptor who starts with a block covered in rough clay and removes a little at each pass until a figure appears. The prompt is the sculptor's brief. Each pass is one denoising step. Working in latent space is like sculpting a small model first and then enlarging it, instead of carving a full-size statue from the start.

What it can do

• Text-to-image generation from a prompt.
• Image-to-image editing, where an input image is partly noised and then denoised toward a new prompt.
• Inpainting, which fills a masked region.
• Outpainting, which extends an image beyond its borders.
• Upscaling and style transfer with suitable variants.
• Conditioned generation with add-ons for edges, depth or pose.

What it is not

• It is not a language model, although it reads text through an encoder.
• It does not understand scenes the way people do. It has learned statistical patterns of how images and words relate.
• It is not a single model. There are several generations and many community variants.

Where it came from

The model grew out of research on diffusion and on latent-space generation. Many pieces came together in the early 2020s.

Foundations

• Early work on diffusion probabilistic models showed that data can be generated by reversing a gradual noising process.
• Denoising diffusion probabilistic models, from 2020, made the approach practical for images and gave high-quality samples.
• Score-based models gave a second view of the same idea, based on the gradient of the data density.
• Faster samplers such as DDIM reduced the number of steps needed.
• Classifier-free guidance gave a simple way to strengthen prompt following.

The latent diffusion step

The 2022 paper on high-resolution image synthesis with latent diffusion models introduced the approach used by Stable Diffusion. It first trained an autoencoder to compress images, then trained the diffusion model in the compressed space. This cut cost sharply while keeping quality.

The open release

An open release of the weights in 2022, by a collaboration of an academic group and companies, made the model widely available. Developers could run it locally, fine-tune it and inspect it. This differed from earlier large text-to-image systems that were available only through hosted services.

Later generations

Later releases changed the architecture, the text encoders and the training recipes, and improved resolution and prompt following. Newer versions replaced the U-Net with transformer-based denoisers in some cases. Because details differ by release, check the specific model card for facts about any version.

Training data

The models were trained on large collections of image-text pairs gathered from the web. This raises open questions about copyright, consent, bias and content safety. These topics remain active in law and policy, and users should review the licence and usage terms of each checkpoint.

How it works

The explanation below follows the life of a generated image, from training to sampling.

Idea 1: noise as a teacher

Take a clean image and add a little random noise. Then add more, step by step, until nothing but static remains. This forward process is easy and needs no learning. The model learns the reverse: given a noisy image and the step number, predict the noise that was added. If you can predict the noise, you can subtract it and move one step closer to a clean image.

Idea 2: work in a small space

A 512 by 512 colour image has about 786,000 numbers. Running many denoising steps over that is costly. The VAE encoder shrinks the image by a factor of eight on each side, giving a latent grid of 64 by 64 with a few channels. That is about 16,000 numbers with four channels, roughly 48 times fewer. The denoising happens there. At the end, the VAE decoder expands the final latent into the full image.

Idea 3: text as guidance

The prompt is split into tokens and encoded into a sequence of vectors by the text encoder. The U-Net uses cross-attention layers. In these layers, image features act as queries, and text vectors act as keys and values. Each spatial location can therefore look at the words that matter for it, such as "red" near a red object.

Step by step: training

1. Take an image and its caption from the dataset.
2. Encode the image into a latent with the VAE encoder.
3. Pick a random time step and add the matching amount of noise to the latent.
4. Encode the caption with the text encoder.
5. Ask the U-Net to predict the noise, given the noisy latent, the step and the text.
6. Compare the prediction with the true noise and update the U-Net.

The VAE and the text encoder are usually frozen. Only the U-Net is trained in the main stage. Captions are sometimes dropped at random so the model also learns to generate without a prompt, which is needed for guidance.

Step by step: sampling

1. Start with random noise in latent space.
2. Encode the prompt, and also an empty prompt.
3. For each step, predict noise twice: with the prompt and without it.
4. Combine the two predictions, pushing away from the empty prompt toward the real prompt.
5. Use the scheduler to subtract a portion of the noise.
6. After the last step, decode the latent into an image with the VAE decoder.

Classifier-free guidance

The guided prediction is the unconditional prediction plus a scale times the difference between the conditional and unconditional predictions. A higher scale follows the prompt more strongly, with less variety and sometimes harsher colours. A lower scale gives more variety and looser matching. Typical values sit in a middle range that you tune by eye.

Schedulers and step counts

The scheduler is the rule for moving between noise levels. Different schedulers trade speed against quality. Some give good images in a few dozen steps, while others need more. Changing the scheduler is a cheap way to speed up generation without retraining.

The U-Net in plain terms

The U-Net has a downward path that shrinks the feature map and an upward path that grows it back. Skip connections carry fine detail across. Inside are residual blocks, self-attention and cross-attention. The shape gives it both a wide view of the whole image and a sharp sense of local detail.

Seeds and reproducibility

The starting noise is generated from a seed. The same seed, prompt, settings and software version usually give the same image. A different seed gives a different image. This is why people share seeds along with prompts.

Image-to-image and inpainting

For image-to-image, the input image is encoded, then partly noised, and denoising begins from that point. A "strength" setting decides how much noise is added, and therefore how far the output may stray. For inpainting, the model keeps the unmasked region fixed at each step and generates only the masked region.

Adapting the model

• Fine-tuning updates the U-Net on a small set of images to teach a style or a subject.
• Low-rank adapters, as in LoRA, train a small add-on that can be shared and switched on or off.
• Textual inversion learns a new token embedding for a concept.
• ControlNet adds spatial control from edges, depth or pose.
• Distillation methods reduce the step count for faster generation.

The forward process in numbers

At step t, the noisy latent is a blend of the clean latent and fresh noise. Early on, the blend is mostly signal. Late in the process, it is mostly noise. A noise schedule sets the blend weights. A simple way to picture it: pour milk into coffee a spoonful at a time. After enough spoonfuls, you cannot tell what the coffee looked like. The model learns to run the film backward.

What the model predicts

Different versions train the network to predict different targets.

• The noise that was added. This is the classic choice.
• The clean latent directly.
• A mix of both, sometimes called a velocity target, which is more stable at extreme noise levels.

All three carry the same information. The choice affects training stability and how well the model behaves at the first and last steps.

Why many steps are needed

Each step makes only a small, local correction, because the model's noise estimate is only accurate for a small change. Big jumps accumulate errors. Better solvers use smarter math to take larger safe steps, so you need fewer of them. Distilled models go further: a student learns to do in a few steps what the teacher does in many.

The VAE in more detail

The VAE is trained first. Its encoder maps an image to a latent grid, and its decoder maps the latent back. Training balances faithful reconstruction against a smooth latent space, plus a perceptual and adversarial term that keeps textures sharp. Because the diffusion model never sees pixels directly, the VAE sets the ceiling on fine detail such as small text and thin lines.

The text encoder in more detail

The text encoder is a pre-trained transformer, often from a contrastive image-text model such as CLIP. It outputs one vector per token. The U-Net reads all of them through cross-attention. Some newer versions use more than one text encoder, or larger language-model encoders, to understand longer and more complex prompts.

Prompt weighting and long prompts

Text encoders have a token limit. Long prompts are cut or split. Tools exist to weight words more or less, or to chunk long prompts. Results differ by tool, so test how your pipeline treats long text.

Model families and formats

Prefer formats that cannot run arbitrary code when loaded, and download weights only from sources you trust.

Key concepts and terms

A worked example

The example uses the Hugging Face Diffusers library. Model identifiers are placeholders, and the current names are found on the model hub.

import torch
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

model_id = "example-org/stable-diffusion-checkpoint"  # placeholder name
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")

generator = torch.Generator(device="cuda").manual_seed(1234)

image = pipe(
    prompt="a clean line drawing of a data center aisle, soft light",
    negative_prompt="blurry, low quality, text",
    num_inference_steps=25,
    guidance_scale=7.0,
    height=512,
    width=512,
    generator=generator,
).images[0]

image.save("example_output.png")

Reading the example

• Half precision halves the memory use of the weights.
• The scheduler swap gives good quality in about 25 steps.
• The generator with a fixed seed makes the result repeatable.
• The negative prompt pushes the image away from unwanted traits.

Image-to-image sketch

from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    model_id, torch_dtype=torch.float16
).to("cuda")
init = Image.open("example_sketch.png").convert("RGB").resize((512, 512))
out = pipe(
    prompt="a watercolor painting of a harbor",
    image=init,
    strength=0.6,
    guidance_scale=7.0,
).images[0]
out.save("example_img2img.png")

Lower strength keeps more of the input. Higher strength gives the model more freedom.

A numeric walk-through of guidance

Say the unconditional noise prediction at some pixel is 0.20, and the prompt-conditioned prediction is 0.30. With a guidance scale of 7, the guided value is 0.20 plus 7 times 0.10, which is 0.90. The model is pushed much further in the direction the prompt suggests. With a scale of 1, the result is just 0.30, the plain conditioned prediction. The numbers are illustrative.

A numeric walk-through of the cost

With 25 steps and guidance, each step needs two U-Net calls, so there are 50 U-Net passes per image. Doing the same on pixel space at full size would be far heavier. This is why the latent trick matters.

Using it for synthetic data

Teams generate images to enlarge training sets for classifiers, test edge cases, or balance rare classes. A simple pipeline is to write prompt templates with controlled variations, generate images, filter them with a classifier or human check, and label them using the prompt. Always validate with real held-out images, since generated images can carry artifacts that the downstream model learns by mistake.

Strengths and limitations

Strengths

• Open weights and a rich ecosystem of tools.
• Runs on consumer hardware, thanks to latent-space operation.
• Flexible: text-to-image, editing, inpainting and conditioned generation.
• Easy to specialise with small adapters.
• Fast iteration, since seeds and settings are controllable.
• Good for synthetic data and design exploration.

Limitations

• Prompt following is imperfect. Counts, spatial relations and negation are hard.
• Rendering legible text inside images is unreliable in many versions.
• Hands, small faces and fine structure can have artifacts.
• Outputs reflect biases in the training data.
• Copyright and consent issues remain open around training data and outputs.
• Misuse risks include misleading images and impersonation.
• Results vary by version, scheduler and settings, which makes comparison hard.
• Sampling takes many steps, which costs time compared with a single forward pass.

Responsible use

• Label generated images clearly when sharing them.
• Do not create images of real people without consent.
• Respect the model licence and its use restrictions.
• Use safety filters where required.
• Keep a record of prompts, seeds and versions for audit.

How it compares with alternatives

How to choose

• Pick Stable Diffusion when you need local control, fine-tuning and open tooling.
• Pick a hosted service when you want quick results without managing hardware.
• Pick a GAN for narrow domains where very fast sampling is vital.
• Pick a conditioned setup with ControlNet when layout must follow a given structure.

Role in tabular foundation models

Stable Diffusion generates pictures, but the ideas behind it reach the world of raw tables in several concrete ways.

Diffusion for tables

The same recipe works on rows. Add noise to a row of numbers and category codes, train a network to remove it, and you have a generative model of tabular data. Numeric columns are noised with Gaussian noise. Categorical columns use a discrete form of diffusion. These models can create synthetic tables that keep the statistical shape of the original while protecting individual records, subject to proper privacy checks.

Synthetic pre-training data

A tabular foundation model such as TabPFN learns from a very wide set of synthetic tables made by random data-generating processes. A diffusion model trained on real tables is another way to make realistic training data. Mixing programmatic synthetic tables, which give coverage and known structure, with learned generative tables, which give realism, is a promising path.

Row and column structure

An image has a fixed grid, so a U-Net fits. A table has rows that are exchangeable and columns that carry different meanings. Table diffusion networks therefore use attention across columns, and often use per-column embeddings, in place of convolutions. Permutation-aware design is important so that row order does not leak into the result.

Mixed types and missing values

Tables mix numbers, categories and dates. A diffusion model must treat each type with a suitable noise process. Missing values are natural to handle in this setting: mask the missing cells and let the model denoise them. This is the same trick as inpainting. Fill the hole while keeping the observed cells fixed. So inpainting in images maps directly to imputation in tables.

No column names

Without headers, a model sees only values and relations. Foundation-style training on many tables pushes the model to learn column roles from data, such as "looks like an identifier" or "looks like a skewed positive amount". Text-conditioned guidance, as used in Stable Diffusion, can be reused. A short description of a column, or a table topic, can guide generation or classification, which supports semantic and sector labelling.

In-context learning

Stable Diffusion adapts through weight updates, such as adapters. Tabular foundation models aim to adapt with no weight updates by placing example rows in the input and attending over them. A diffusion generator can feed that process by supplying extra, plausible example rows for rare classes. Check these rows carefully, since false patterns can mislead the in-context learner.

Continual learning

Adapters and low-rank add-ons, a pattern strongly established in the Stable Diffusion community, give a practical template for continual learning. A new domain gets a new small module. Old modules stay unchanged. The same pattern helps tabular models take on new industries without forgetting previous ones.

Anomaly detection

A diffusion model trained on normal rows learns what normal looks like. Rows that are hard to denoise, or that the model reconstructs poorly, can be flagged as unusual. The reconstruction error acts as an anomaly score. This resembles how autoencoders spot outliers, with a stronger generative model behind it.

Caveats

• Synthetic tables can leak private records if the model memorises them. Test for this.
• Generated rows may violate business rules, such as an end date before a start date. Add constraints and checks.
• Evaluate usefulness by training on synthetic data and testing on real held-out data.

See /model/architecture for how a table model handles structure, /model/research for context, and /model/benchmarks for measured results.

Practical guidance and best practices

Prompting

• Start with a clear subject, then add style, lighting and composition.
• Use short, concrete phrases. Long lists of adjectives often add noise.
• Use negative prompts to remove common faults, such as blur or extra limbs.
• Change one thing at a time when you tune a prompt.
• Keep a prompt log with seeds and settings.

Settings

• 20 to 30 steps are often enough with a modern scheduler.
• A guidance scale in a middle range is a safe starting point.
• Generate at the native training resolution, then upscale. Very different sizes can cause repeated objects.
• Use half precision and memory-saving options to fit smaller GPUs.

Fine-tuning

• For a style or subject, use a low-rank adapter. It trains fast and is easy to share.
• Use 10 to 30 clean, varied images of a subject, with consistent captions.
• Watch for overfitting: outputs start to copy training images.
• Keep a test prompt set to compare checkpoints.
• Combine with PEFT tooling when you want a standard adapter workflow.

Conditioning and control

• Use ControlNet when layout, pose or depth must be fixed.
• Use image-to-image with a low strength to refine a draft.
• Use inpainting for local fixes, not for whole-image changes.

Production

• Queue requests and batch where possible.
• Set limits on resolution and steps to control cost.
• Add content filters and logging.
• Pin software versions so images stay reproducible.
• Consider exporting to an optimised runtime such as ONNX Runtime for speed.

Evaluating output

• Combine human review with simple automatic checks.
• Track a fixed set of prompts over time.
• For synthetic data, measure downstream task accuracy on real data.
• Check diversity as well as quality, since repeated near-duplicates add little.

Common mistakes

• Cranking the guidance scale too high, which gives oversaturated, rigid images.
• Using too few steps with a slow scheduler.
• Generating at a very large size directly and getting duplicated objects.
• Expecting exact text, counts or spatial layout from the prompt alone.
• Fine-tuning on too few or too similar images until the model memorises them.
• Mixing incompatible checkpoints and adapters from different base versions.
• Forgetting to pin the seed and version when you need repeatability.
• Ignoring licence terms for the base model, adapters and training images.
• Using generated images as training data without checking for artifacts.
• Assuming generated tabular data is automatically private. Test it.

Troubleshooting checklist

1. Is the output a blank or black image? Check the safety filter and the precision settings.
2. Are faces or hands wrong? Try more steps, a different scheduler or inpainting.
3. Is the image blurry? Raise the step count or check that the VAE matches the checkpoint.
4. Does the style ignore your prompt? Adjust the guidance scale and prompt wording.
5. Out of memory? Reduce size, use half precision, or enable attention slicing.

Everyday workflows

1. Concept art: generate many low-cost drafts, pick one, refine with image-to-image.
2. Product mock-ups: use inpainting to change a background while keeping the product.
3. Dataset building: write prompt templates, generate, filter and label.
4. Education: illustrate abstract ideas with consistent style adapters.
5. Research: study how prompts, seeds and schedulers change outputs.

Hardware notes

• A GPU with a moderate amount of memory is enough for 512-pixel generation in half precision.
• Larger images and newer, bigger models need more memory.
• Memory-saving options include attention slicing, tiled VAE decoding and offloading to CPU.
• CPU-only runs work but are slow.
• Compiled or exported models can speed up inference.

Frequently asked questions

What is Stable Diffusion?

Stable Diffusion is an open text-to-image model based on latent diffusion. It turns a text prompt into an image by starting from random noise in a compressed latent space and removing noise step by step, guided by the prompt, then decoding the result into pixels.

Why does it work in latent space?

Latent space is much smaller than pixel space, so each denoising step is far cheaper. A VAE compresses the image and later expands it back, which keeps most visual detail while cutting compute. This is why the model can run on a single consumer GPU.

What is classifier-free guidance?

Classifier-free guidance is a method that makes the image follow the prompt more closely. The model predicts noise with and without the prompt and then pushes the result away from the unprompted prediction. A higher guidance scale means stronger prompt following but less variety.

What does the U-Net do?

The U-Net is the denoising network. At each step it looks at the noisy latent, the step number and the prompt vectors, and predicts the noise to remove. Its shrink-and-grow shape with skip connections lets it use both global context and local detail.

How do I get the same image twice?

Use the same prompt, seed, settings, scheduler and software version. The seed fixes the initial noise, so the process becomes repeatable. Small changes in hardware or library versions can still cause slight differences.

Can I train it on my own images?

Yes. Common approaches include fine-tuning, low-rank adapters and textual inversion. Adapters are popular because they train quickly, use little memory and can be shared as small files. Use clean, varied images and watch for overfitting.

Is Stable Diffusion good for creating tabular data?

The base model makes images, not tables. The diffusion idea, however, has been adapted to rows of mixed numeric and categorical data, and such models can create synthetic tables. Always validate privacy, business rules and usefulness on real held-out data.

Why are hands and text often wrong?

The model learns visual statistics, not a physical or symbolic understanding. Hands have many small parts in varied poses, and text needs exact letter shapes. Both are hard to get right from patterns alone, though newer versions improve on them.

What is the difference between Stable Diffusion and a GAN?

A GAN trains a generator against a discriminator and samples in one pass, which is fast but can be unstable and less diverse. Stable Diffusion trains a denoiser and samples over many steps, which is slower but more stable and gives wide variety plus text control.

Is it legal to use generated images commercially?

It depends on the model licence, the training data questions and local law, which are still changing. Read the licence for the checkpoint and any adapters, and get legal advice for commercial projects.

Glossary

• Adapter: A small trainable module added to a large model.
• Autoencoder: A network that compresses data and then rebuilds it.
• Cross-attention: Attention from one sequence to another, used to read the prompt.
• Denoising: Removing noise from data.
• Diffusion: A process of gradually adding noise, and learning to reverse it.
• Guidance: A technique that steers generation toward a condition.
• Inpainting: Filling in a masked region.
• Latent: A compact internal representation of data.
• Negative prompt: Text describing what to avoid.
• Noise schedule: The plan for how much noise is present at each step.
• Prompt: The text that describes the desired image.
• Sampler: Another name for the scheduler or step rule.
• Seed: The number that sets the random starting noise.
• Synthetic data: Artificially generated data used for training or testing.
• Textual inversion: Learning a new word embedding for a concept.

Summary

Stable Diffusion is an open latent diffusion model that generates images from text. A VAE compresses images, a U-Net learns to remove noise in that compressed space, and a text encoder guides the process through cross-attention. Classifier-free guidance, schedulers and seeds give control over quality, variety and repeatability.

It is efficient, flexible and easy to adapt with small modules, but it has limits in prompt precision, fine detail and legal clarity. For tabular work, its ideas carry over: denoising can generate and impute table rows, inpainting maps to imputation, adapters support continual learning, and reconstruction error supports anomaly detection. Related reading includes ControlNet, Diffusion models, CLIP, TabPFN and /model/research.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: