Perceiver IO

Overview

Perceiver IO is a neural network architecture that reads a large input through a small, fixed-size set of learned vectors called a latent array, and then decodes any kind of output from that array. The input can be pixels, words, audio samples, table cells or a mix of them. The output can be a single label, a sequence, a full image or one value per query. The central trick is cross-attention: a small number of latent vectors ask questions of a big input, and a set of output queries ask questions of the latents.

The reason it matters is cost. Standard self-attention grows with the square of the input size, which becomes painful for long or large inputs. In Perceiver IO the heavy attention happens in the small latent space, so the cost grows only linearly with the input size. For tabular data this gives a global reasoning path that does not blow up as a table gains rows and columns. It pairs well with axial attention, which handles local row and column structure, and it fits inside the larger design of a tabular foundation model.

What it is

Perceiver IO is a general-purpose architecture introduced in 2021 as the successor to the original Perceiver. The name stands for a model that can take Inputs of any shape and produce Outputs of any shape ("IO"). It was designed to avoid building a new network for each data type.

The architecture has three parts.

1. Encoder. One cross-attention step moves information from the large input array into the small latent array.
2. Processor. A stack of ordinary self-attention layers works only on the latent array.
3. Decoder. A cross-attention step lets a set of output queries read from the latent array to produce outputs of the desired size and kind.

Why it is different from a plain transformer

A plain transformer uses the input positions as its working space. If you feed in 10,000 tokens, every layer deals with 10,000 positions. In Perceiver IO the working space is the latent array, which might have only a few hundred vectors regardless of how large the input is. The input is touched once at the start, and the output is produced by queries at the end.

Why "IO"

The original Perceiver could take large inputs but could only produce a single output vector, such as a class. Perceiver IO added a flexible decoder. By choosing the output queries, you choose the shape of the output. One query gives one label. Ten thousand queries give ten thousand outputs, such as one per pixel or one per masked table cell.

Where it came from

The story starts with the observation that most successful architectures are specialized. Convolutional networks suit images, recurrent networks and transformers suit sequences, graph networks suit graphs. Each makes assumptions about the data. Researchers asked whether one architecture could handle many kinds of data without hand-built assumptions.

The Perceiver, presented in 2021 in a paper titled "Perceiver: General Perception with Iterative Attention", answered with a latent bottleneck. A small latent array repeatedly cross-attends to the raw input, which could be flattened pixels or other signals. That removed the quadratic cost on the input but the model produced only a classification-style output.

Perceiver IO, described in the paper "Perceiver IO: A General Architecture for Structured Inputs and Outputs" (2021, published at a conference in 2022), added the query-based decoder. It showed results across language, optical flow, images, audio-video tasks and games, using almost the same design. A well-known demonstration was a language model that reads raw bytes instead of word pieces, which removes the need for a tokenizer.

Ideas it built on

• Attention and the transformer (2017).
• Cross-attention, as used in encoder-decoder models.
• Learned queries, as in set-based and detection models. See DETR.
• Inducing points from set transformers, a similar idea of summarizing a set by a small learned set.
• Fourier and learned position features to tell the model where an input element sits.

Why tabular researchers care

Tables have a flexible shape: variable numbers of rows and columns, and cells of many types. Perceiver IO has no fixed input size and no built-in grid assumption. A table can be flattened into a list of cells, each with a value embedding and a position or identity embedding, and the architecture accepts it. That flexibility, plus the linear cost, made it a candidate for foundation models trained on many table shapes. Later sections show how it is used.

The core intuition

Most data is redundant. A photo of a street has millions of pixels, yet a handful of facts describe it. A table of fifty thousand cells often follows a few patterns: scales of columns, relations between columns, groups of similar rows. A good summary is much smaller than the raw data. Perceiver IO builds that summary explicitly and does all its serious thinking on the summary.

Analogy: a news editor does not reread every wire story before each meeting. Reporters condense the day into a one-page brief. The editor discusses the brief, and then answers specific questions by looking things up in it.

What changed from the first Perceiver

Design choices that matter

• Latents are parameters. They are learned once and reused for all inputs, then specialize through attention to each input.
• Queries are inputs to the decoder. They can be learned, built from positions, or built from table identities.
• Layer normalization and residual links are used throughout, as in a standard transformer.
• Multi-head cross-attention lets different heads pick up different kinds of relations.

How it works

Follow one forward pass step by step, with an everyday analogy for each step.

Step 1: Prepare the input array

The raw input becomes an array of vectors. For an image, each pixel becomes a vector. For text, each byte or token becomes a vector. For a table, each cell becomes a vector made of a value embedding plus an identity embedding that says which column and which row it belongs to.

Position information is added because the architecture itself does not know where elements come from. Common choices are Fourier features or learned embeddings.

Analogy: a library receives thousands of loose pages. Each page is stamped with a label saying where it came from.

Step 2: Create the latent array

The latent array is a small set of learned vectors, for example 256 or 512 of them. They are parameters of the network, so they start as the same vectors for every input. Think of them as a team of reviewers with empty notebooks.

Analogy: ten reviewers sit in a room. They have not seen the pages yet, but each has a different habit of what to look for.

Step 3: Encode with cross-attention

The latent vectors act as queries. The input vectors act as keys and values. Each latent asks "which parts of the input matter for me?" and writes what it finds into its notebook. The cost is the number of latents times the number of inputs, which is linear in the input size for a fixed latent count.

Analogy: each reviewer skims all the pages and takes notes on what interests them.

Step 4: Process with self-attention

The latent array now passes through a stack of self-attention layers. The latents talk to each other, refine their notes and build a shared understanding. This stack can be deep because the array is small.

Analogy: the reviewers hold a meeting and compare notes. They can hold many meetings, since there are only a few of them.

Step 5: Decode with query attention

A set of output queries is built. Each query says what the user wants: "the label", "the value at this pixel", "the missing cell in row 7, column 3". The queries cross-attend to the latents and each receives a vector, which a small head turns into the final output.

Analogy: the user asks specific questions, and the reviewers answer from their shared notes.

Step 6: Optionally repeat

The original Perceiver allowed the encoder cross-attention to be repeated with shared weights, giving the latents more chances to look at the input. Perceiver IO usually encodes once and spends the depth in the latent processor.

Cost comparison

Let N be the number of input elements, M the number of latents, O the number of output queries and L the number of processor layers.

Since M is fixed and small, the total grows linearly with N and O. A plain transformer pays N squared at every layer.

A compact code sketch

import torch
import torch.nn as nn

class CrossAttend(nn.Module):
    def __init__(self, q_dim, kv_dim, heads=4):
        super().__init__()
        self.attn = nn.MultiheadAttention(q_dim, heads, kdim=kv_dim,
                                          vdim=kv_dim, batch_first=True)
        self.norm_q = nn.LayerNorm(q_dim)
        self.norm_kv = nn.LayerNorm(kv_dim)
        self.ff = nn.Sequential(nn.Linear(q_dim, 4 * q_dim), nn.GELU(),
                                nn.Linear(4 * q_dim, q_dim))

    def forward(self, q, kv):
        out, _ = self.attn(self.norm_q(q), self.norm_kv(kv), self.norm_kv(kv))
        q = q + out
        return q + self.ff(q)

class TinyPerceiverIO(nn.Module):
    def __init__(self, in_dim=32, lat_dim=64, n_lat=64, out_dim=1, depth=4):
        super().__init__()
        self.latents = nn.Parameter(torch.randn(n_lat, lat_dim) * 0.02)
        self.encode = CrossAttend(lat_dim, in_dim)
        self.process = nn.ModuleList([
            nn.TransformerEncoderLayer(lat_dim, 4, 4 * lat_dim,
                                       batch_first=True, norm_first=True)
            for _ in range(depth)])
        self.decode = CrossAttend(in_dim, lat_dim)   # queries live in in_dim
        self.head = nn.Linear(in_dim, out_dim)

    def forward(self, x, queries):
        # x: (batch, N, in_dim)   queries: (batch, O, in_dim)
        z = self.latents.unsqueeze(0).expand(x.size(0), -1, -1)
        z = self.encode(z, x)
        for layer in self.process:
            z = layer(z)
        return self.head(self.decode(queries, z))

x = torch.randn(2, 5000, 32)      # 5000 input elements
q = torch.randn(2, 10, 32)        # 10 output queries
print(TinyPerceiverIO()(x, q).shape)   # torch.Size([2, 10, 1])

The key point is that the 5,000 input elements are touched only by one cross-attention, and the ten output queries touch only the 64 latents.

Variations of the output queries

The output side is where most of the flexibility lives. Here are common ways to build queries.

• A single learned query. Used for one label per input, such as a table-level class.
• One query per position. Used for dense outputs such as one value per pixel.
• One query per target cell. Built from the row and column identity of the cell to predict.
• One query per column. Used to predict a type or a summary for every column.
• Queries that mix input features. A query may carry part of the input, for example the features of a test row, so that the output depends on it directly.

In every case, the decoder works the same way. The query reads the latents and returns a vector.

Why separating the thinking from the reading helps

When reading and thinking are mixed in one stack, every layer must carry the full input along. When they are separated, reading happens once and thinking happens in a compact space. The same split is useful in engineering: a loader reads big files once and a worker thinks about a compact index.

Key concepts and terms

A worked example

Suppose we want to predict a missing numeric cell in a table with 2,000 rows and 20 columns.

Setup

• The table has 40,000 cells. Each observed cell becomes a vector: a value embedding plus a column identity embedding plus a row-group embedding.
• We use 128 latent vectors of size 64.
• We want one output: the value of the missing cell at row 5, column 7.

Walk-through

1. Flatten. The 40,000 cell vectors (minus the one missing) form the input array. The missing cell is not in the input.
2. Encode. The 128 latents cross-attend to the 40,000 cells. That is 128 times 40,000, about 5.1 million pairs. A full self-attention over 40,000 cells would be 1.6 billion pairs, about 300 times more.
3. Process. Six self-attention layers on 128 latents cost 6 times 128 squared, about 98,000 pairs. This is tiny.
4. Decode. One output query is created. It carries the identity of column 7 and the row identity of row 5. It cross-attends to the 128 latents, which costs 128 pairs.
5. Head. A small linear layer turns the resulting vector into a number.

What the latents learn to do

During pre-training, different latents tend to specialize. One may gather the overall scale of a column. Another may track correlations between two columns. Another may focus on the rows that look similar to the query row. The query then selects which of those summaries is helpful for its target.

Comparing the cost for several sizes

The ratio grows in proportion to the input size. That is the whole benefit of the bottleneck.

What you give up

The 128 latents must carry everything the output needs. If the table holds very detailed information, such as thousands of independent columns that all matter, the latents may become a bottleneck for quality. The fix is to use more latents, more depth, or to combine the latent path with local attention such as axial attention.

Checking the arithmetic on memory

Memory matters as much as time. The attention score matrix in cross-attention has size M by N per head. With 256 latents and 100,000 input elements that is 25.6 million scores per head, which fits easily. The self-attention matrix of a plain transformer on 100,000 elements would hold ten billion scores per head, which does not fit without special kernels. The latent design therefore also removes a memory wall.

Input streaming

Because the encoder is a single cross-attention, the input can be processed in chunks. The latents can accumulate evidence chunk by chunk, using a running softmax. This allows tables larger than device memory to be read in parts. It is an engineering option rather than a core feature, but it follows directly from the design.

Strengths and limitations

Strengths

• Linear cost in input size. Inputs with tens of thousands of elements are manageable.
• Input-agnostic. The same network accepts images, text, audio, tables or combinations.
• Output-agnostic. The query decoder produces any number of outputs of any kind.
• Few assumptions. No built-in grid or sequence prior is needed, which suits irregular data such as tables.
• Deep latent processing. Depth is cheap because the latent array is small.
• Natural multi-task design. Different heads and queries can share one latent summary.
• Byte-level modelling. It can remove the need for a separate tokenizer.

Limitations

• Compression loss. A small latent array can drop fine detail.
• Weaker inductive bias. Without grid or locality priors, it may need more data or good position features to match specialized models on images.
• Position features matter. Poor position or identity encodings hurt a lot.
• Training cost. Although each pass is cheap, strong results often need large-scale training.
• Latent count is a hyperparameter. Too few latents limit quality; too many lose the saving.
• Interpretability. Because information passes through a compressed summary, tracing a prediction to an input cell is harder.

How it compares with alternatives

Perceiver IO and axial attention together

These two ideas are complementary. Axial attention keeps the table structure explicit and gives each cell rich context from its row and column. Perceiver IO provides a global summary that does not depend on grid shape. Some designs run axial layers first and then pool into latents, or they feed latents back into the axial layers as extra global tokens.

Role in tabular foundation models

A tabular foundation model learns from raw tables with mixed types, missing values and often no column names, and must serve many unseen datasets. Perceiver IO offers three useful things for that goal: shape freedom, a linear cost, and query-based outputs.

Shape freedom for rows and columns

Tables differ in width and length. A flattened array of cells with identity embeddings can represent any of them. The latent array is always the same size, so the downstream layers do not care about the original shape. This is what lets one set of weights serve a table with five columns and a table with five hundred.

Mixed types and missing values

Numeric cells use a numeric encoder, categorical cells use embeddings, and dates or text use their own encoders. All end up as vectors of the same size. Missing cells are simply left out of the input array or given a missing embedding. Because there is no fixed grid, leaving cells out is natural.

No column names

The identity embedding of a column can be learned from scratch or derived from the column's data, such as summary statistics of its values. If a header exists, its text can be embedded and added. If not, the model still works from the data patterns, which matches real-world tables with cryptic headers.

Pre-training on synthetic and real tables

Pre-training uses many generated tables, which differ widely in shape, plus real tables. A shape-free architecture is convenient here. The cost stays low even for big synthetic tables, so more data can be seen per training hour. See synthetic data generation and /model/research.

In-context learning

For in-context prediction, labeled rows and unlabeled query rows are both placed in the input array. The latents gather information from the labeled rows. The output queries, one per query row, read the answers from the latents. This gives a route to in-context learning that scales to many context rows without comparing every row to every other row. It complements the row-to-row attention used by TabPFN and TabICL.

Masked cell modeling and imputation

Perceiver IO is a natural host for masked cell modeling. The masked cells are not in the input. Instead, one output query per masked cell is created, containing the cell's column and row identity. The decoder returns a value for each query. This is exactly how the architecture handles "one output per query". It is also the basis for imputation networks that fill missing cells and return a confidence.

Continual learning

The latent processor is a compact shared core. New table families can be absorbed by fine-tuning small input encoders or output heads while keeping the core mostly fixed, and protection methods such as elastic weight consolidation or learning without forgetting can guard the core. The small size of the core makes such protection cheaper.

Anomaly detection

A model trained to reconstruct cells from the latent summary gives a reconstruction error per cell. Cells with high error are suspicious. Because the latents summarize the whole table, they capture global norms, which helps identify values that are odd in the context of the full table. See anomaly detection.

Sector and semantic classification

Deciding what a table is about, or what type a column has, needs a view of the whole table. A single output query with a class head reads the latents and predicts a sector or a type. Many queries, one per column, can predict a semantic type for each column. See semantic and sector heads. Results across tasks are collected at /model/benchmarks, and the design principles are described at /model/architecture.

Using the latent summary directly

The latent array after the processor is a compact description of the table. It can be reused.

• Store it as a fingerprint of the table and compare tables by it.
• Feed it to a small classifier for table-level labels.
• Use it as a conditioning signal for generating synthetic rows.
• Cache it, so that many output queries over the same table cost only the decoder.

This caching is useful in production: encode a table once, then answer many cell-level questions cheaply.

Evaluation plan

1. Split by table, not by row, to test generalization to unseen tables.
2. Report quality by table size and by missing-value rate.
3. Compare latent counts to find the knee of the curve.
4. Measure latency for the encode step and the decode step separately.
5. Include a gradient-boosting baseline and a simple statistical baseline such as column means.

Practical guidance and best practices

• Pick the latent count with the task in mind. Start with a few hundred latents. Increase if validation quality is limited by detail.
• Give every cell an identity. Column and row-group embeddings are what make a flattened table readable.
• Use several decoder queries for rich outputs. One query per target cell, per column or per row, depending on the task.
• Add a global query for table-level labels. A single learned query can summarize the table for sector or domain classification.
• Combine with local structure. Add axial attention before or after the latent path when row and column structure matters.
• Normalize inputs. Scale numeric columns robustly and encode categorical values consistently.
• Mask carefully. In in-context setups, make sure query labels are never in the input.
• Use mixed precision and fused kernels. See mixed precision and FlashAttention.
• Warm up and clip. Use learning-rate warm-up and gradient clipping, since cross-attention from random latents can be unstable early on.
• Check calibration. Models that output probabilities or intervals should be checked on held-out tables.
• Benchmark against a boosted tree. Use XGBoost or LightGBM as a reality check.

Choosing parameters

These numbers are starting points, not rules.

Training recipe in brief

1. Build a sampler that draws tables of many shapes.
2. Choose masks: random cells, whole columns, whole rows, and structured patterns.
3. Feed the visible cells as input and create queries for the hidden ones.
4. Train with a loss suited to each type: squared error or a distribution loss for numbers, cross-entropy for categories.
5. Evaluate on tables of shapes and types not seen in training.

Notes on training stability

• Initialize latents with small values so early cross-attention is not dominated by noise.
• Use pre-norm layers, which are easier to train at depth.
• Lower the learning rate for the latent parameters if they drift quickly.
• Use dropout on the input elements, which acts as free data augmentation and also teaches robustness to missing cells.
• Monitor the attention entropy of the encoder, since a collapse to a few input elements signals a problem.

Common mistakes

• Too few latents. The summary becomes too small to hold what the output needs.
• Missing identity features. Without column and row identity, a flattened table is just a bag of numbers.
• Leaking targets into the input. Masked cells must be removed, not zeroed, if zero is a meaningful value.
• Ignoring local structure. Treating a table as a bag of cells can lose the benefits of explicit rows and columns.
• Expecting free lunch on images. Without priors, a Perceiver may need much more data than a CNN or ViT for pictures.
• Skipping normalization of mixed units. Columns with vastly different scales confuse the encoder.
• Overfitting one table shape. Test on unseen widths and lengths.
• Reading latents as explanations. Latents are learned summaries, not human-readable features.
• Unstable early training. Skipping warm-up or clipping often leads to loss spikes.

Quick reference: when to reach for Perceiver IO

Frequently asked questions

What is Perceiver IO in simple terms?

Perceiver IO is a neural network that reads a large input through a small set of learned vectors called latents, thinks using only those latents, and then answers questions about the input through a set of output queries. Because the heavy thinking happens in the small latent space, the cost grows slowly even when the input is very large.

What does the IO in Perceiver IO stand for?

It stands for inputs and outputs. The original Perceiver could take large inputs but produced one output vector, which limited it to tasks like classification. Perceiver IO added a flexible query-based decoder, so the output can be any size and kind, such as one value per pixel, one label per token or one value per masked table cell.

How is Perceiver IO different from a standard transformer?

A standard transformer applies self-attention across all input positions at every layer, so cost grows with the square of the input size. Perceiver IO applies cross-attention from a small latent array to the input once, then uses self-attention only on the latents. Its cost grows linearly with input size and its output shape is chosen by the queries.

Why is Perceiver IO useful for tabular data?

Tables vary in shape, have mixed cell types and often have missing values. Perceiver IO accepts any set of cells as input, uses a fixed-size latent summary regardless of table shape, and can return an output for any chosen cell through a query. This makes it a flexible and affordable global reasoning path for models trained across many tables.

How does Perceiver IO handle missing cells?

Missing cells can simply be left out of the input array, because the architecture does not need a full grid. To fill a missing cell, an output query that names its row and column is created, and the decoder returns an estimate from the latent summary. This is the same mechanism used in masked cell pre-training.

Does the latent bottleneck lose information?

It can, because the latents are a compressed summary. The loss is controlled by the number and size of the latents and the depth of the processor. For many tasks a few hundred latents are enough. When fine detail matters, more latents or additional local attention, such as along rows and columns, can recover it.

Can Perceiver IO be combined with axial attention?

Yes, and the combination is natural. Axial attention provides exact local context along rows and columns, while the latent path provides a cheap global summary. A model can apply axial layers to the cell grid and then pool into latents, or feed latent vectors back as extra tokens, getting the strengths of both.

Does Perceiver IO work for images and text too?

Yes. It was demonstrated on images, language, audio, video and game tasks with nearly the same design. For text it can read raw bytes without a tokenizer. For images it can work from pixels, though specialized models with locality priors may reach a given accuracy with less data.

How many latents should I use?

There is no universal answer. A few dozen can be enough for simple tasks, a few hundred for typical tabular models, and a thousand or more for rich multi-task settings. Start modestly, watch validation quality, and increase the count only if the model appears limited by the amount of information it can hold.

Is Perceiver IO better than gradient boosting on tables?

Not automatically. Gradient boosting remains a strong and fast baseline on single tables. Perceiver IO is attractive when you want one pre-trained model that handles many table shapes, supports masked prediction and in-context learning, and scales to large inputs. Comparing both on your own data is the best way to decide.

Glossary

• Bottleneck: a deliberately narrow stage that forces the model to summarize.
• Byte-level model: a model that reads raw bytes rather than tokens.
• Cross-attention: attention in which queries come from one array and keys and values from another.
• Decoder query: a vector that specifies which output the model should produce.
• Fourier features: sine and cosine encodings of position at several frequencies.
• Identity embedding: a learned vector that tells the model which column or row a cell belongs to.
• In-context learning: adapting to a task from examples included in the input.
• Latent array: the small learned set of vectors that holds the working summary.
• Linear scaling: cost that grows in proportion to input size.
• Masked cell modeling: training by hiding cells and predicting them.
• Modality: a kind of data such as image, text, audio or table.
• Processor: the stack of self-attention layers acting on the latents.
• Query-based decoding: producing outputs by letting queries attend to the latents.
• Self-attention: attention within a single array.
• Set Transformer: a model for sets that uses learned inducing points to summarize inputs.
• Tabular foundation model: a model pre-trained on many tables that serves new tables with little or no training.
• Attention head: one of several parallel attention computations in a layer.
• Chunking: processing a large input in smaller pieces.
• Dense prediction: producing an output for every position, such as every pixel.
• Dropout: randomly hiding units or inputs during training to improve robustness.
• Fingerprint: a compact vector that describes a table.
• Inducing points: a small learned set used to summarize a larger set.
• Layer normalization: rescaling activations to stabilize training.
• Pre-norm: applying normalization before each sublayer.
• Residual connection: a shortcut that adds a layer input to its output.
• Run-time cache: stored intermediate results reused across queries.
• Weight sharing: using the same parameters in several layers.

Summary

Perceiver IO reads a large input through a small learned latent array using cross-attention, reasons with self-attention on the latents only, and decodes any output through query-based cross-attention. Its cost grows linearly with the input size, and its input and output shapes are flexible.

For tabular foundation models this means a global reasoning path that scales to large tables, accepts any shape, tolerates missing cells and returns outputs for exactly the cells or labels you ask for. It works well with masked cell modeling, supports imputation networks and anomaly scoring, and complements axial attention. Its main risk is compression loss in the latent array, which is managed through latent count, depth and a hybrid design. For wider context see /model/architecture, /model/research and /model/benchmarks.

Key takeaways

• A small latent array reads the input once and does the thinking.
• Output queries decide what comes out and how many outputs there are.
• Cost is linear in input size and memory stays small.
• Tables of any shape fit as lists of identity-tagged cells.
• Masked cell queries make imputation and pre-training natural.
• Combine with local row and column attention for the best of both.
• Tune latent count and depth against validation quality.

Related

It belongs to Tabular foundation models in my stack, under Foundation models and applied ML. Others in the same group: