# Neural network architectures: MLPs, CNNs, RNNs, transformers, MoE and diffusion

An architecture is a set of assumptions about the data built into the network. This page compares the main families and what each assumes.

## Overview

Every architecture trades generality for built-in knowledge. A convolution assumes nearby pixels belong together; a recurrent network assumes order; a transformer assumes any part of the input may matter to any other and learns which. The right assumptions save data and compute.

## The families

| Architecture | Assumption | Typical use | Read more |
|---|---|---|---|
| MLP | None beyond layers | Tabular data, small problems, parts of larger models | [Neural networks](/tools/neural-networks) |
| CNN | Local patterns repeat across space | Images, audio, some time series | [CNN](/tools/cnn), [ResNet](/tools/resnet) |
| RNN, LSTM, GRU | Order matters, read step by step | Older sequence models, small devices | [RNN](/tools/recurrent-neural-networks), [LSTM](/tools/lstm) |
| Transformer | Any token can attend to any other | Language, vision, audio, tables | [Transformers](/tools/transformers), [attention](/tools/attention-mechanism) |
| Mixture of experts | Each token needs only some of the network | Large language models | [MoE](/tools/mixture-of-experts) |
| State-space models | Long sequences with linear cost | Long context, audio | [Mamba](/tools/mamba), [S4](/tools/s4) |
| Diffusion models | Generate by removing noise step by step | Images, video, and now text | [Diffusion](/tools/diffusion-models), [diffusion LMs](/tools/diffusion-language-models) |
| Graph networks | Data is nodes and edges | Molecules, networks, recommendations | [GCN](/tools/gcn), [GAT](/tools/gat) |

## Why transformers took over

Attention lets every token gather information from every other token in one layer, and it runs in parallel on GPUs. With enough data, that flexibility beats built-in assumptions, which is why transformers now dominate language and are strong in vision ([ViT](/tools/vit)) and tables ([tabular foundation models](/tools/tabular-foundation-models)). Their cost grows with the square of the sequence length, which drives work on [FlashAttention](/tools/flashattention), the [KV cache](/tools/kv-cache) and state-space alternatives.

## Sparse and parallel generation

Mixture-of-experts layers route each token to a few of many expert sub-networks, so a model can have hundreds of billions of parameters while each token uses only a fraction. Diffusion language models, such as DiffusionGemma released in June 2026, generate a block of text in parallel over a few refinement steps instead of one token at a time.

## Learn it hands-on

Try [convolution](/ai-simulations/convolution), [attention](/ai-simulations/attention), [positional encoding](/ai-simulations/positional-encoding) and [mixture of experts](/ai-simulations/mixture-of-experts). The [GPT architecture](/tools/gpt-architecture) page walks through a full decoder.

## Frequently asked questions

### Are CNNs obsolete?

No. They remain efficient for many vision tasks and edge devices; modern designs such as [ConvNeXt](/tools/convnext) compete with vision transformers.

### What replaces attention for very long inputs?

State-space models, linear attention variants and memory architectures such as [Titans and MIRAS](/articles/transformer-2-titans-miras) are the main candidates.

## Summary

Choose an architecture by the structure of the data: convolutions for local patterns, transformers for flexible relations, mixture of experts for scale, state-space models for long sequences, and diffusion for parallel generation.

## Back to the topic

This page is part of **[Deep learning](/topics/deep-learning)**, which maps every concept, tool, simulation and article on the subject.
