GAT

Overview

A GAT, or graph attention network, is a graph neural network that learns how much each neighbour should matter to a node. Instead of taking a fixed average of neighbours like GCN, it computes an attention score for every edge, turns the scores into weights, and takes a weighted sum of neighbour vectors. Several attention heads run in parallel, and their outputs are combined.

This makes GAT a good fit when neighbours are not equally useful. In a network of accounts, a few links may carry most of the signal. In a molecule, some bonds matter more than others. Learned weights also give a partial view into what the model relies on. For tabular work, GAT is one of the standard ways to bring relational context into a model, and it shares its core mechanism with the attention used in transformers and in models that read rows and columns of tables.

What it is

GAT is a message passing network whose aggregation step is self-attention restricted to graph neighbours. Each node looks only at the nodes it is linked to, scores them, and mixes their features by those scores.

Inputs and outputs

The input is a graph with node features, and optionally edge features in extended versions. The output is a new vector per node. As with GCN, those vectors can feed a node classifier, a graph-level pooling step or a link predictor.

Masked attention

In a transformer, every token may attend to every other token. In GAT, attention is masked by the graph. A node can attend only to its neighbours and itself. This keeps cost proportional to the number of edges instead of the square of the number of nodes.

Why attention rather than a fixed average

A fixed average assumes all neighbours are equally informative, up to degree scaling. Real graphs rarely work that way. Attention lets the data decide. It also makes the layer independent of the global graph structure, since it needs only local neighbourhoods. That helps with graphs not seen during training.

Where it came from

Attention first became popular in sequence models for translation, where a decoder learned to focus on relevant input words. The transformer then made attention the main building block. Graph neural networks, meanwhile, had mostly used fixed averaging or spectral filters, as in the GCN.

The paper "Graph Attention Networks" appeared in 2018. It proposed masked self-attention over neighbourhoods, multi-head attention for stability, and a design that works both when the whole graph is available and when new graphs appear at test time. It matched or improved on strong baselines for citation networks and an inductive protein task.

A later paper introduced GATv2, which showed that the original GAT computes a kind of static attention in which the ranking of neighbours does not depend on the query node. GATv2 changes the order of operations to allow dynamic attention. Many later graph transformers build on the same ideas. See /model/research for how related attention designs are studied for tables.

How it works

Imagine a group project where you need advice from teammates. You do not weigh every teammate's opinion equally. You glance at each person, judge how relevant they are to your current question, and then weigh their input by that judgement. GAT does this for every node at once.

Step 1: Project features

Each node vector h_i is multiplied by a shared weight matrix W, giving z_i = W h_i. This puts all nodes in a common space and lets the model change the width.

Step 2: Score each edge

For a node i and a neighbour j, GAT builds a raw score. In the original form, the two projected vectors are concatenated and multiplied by a learned vector a, then passed through a leaky ReLU:

e_ij = LeakyReLU( a^T [ z_i || z_j ] )

The symbol || means concatenation. The score says how useful j looks to i.

Analogy: a quick gut feeling about each teammate, computed from both the asker and the teammate.

Step 3: Normalise with softmax

For each node i, the scores over its neighbourhood are normalised with softmax:

alpha_ij = exp(e_ij) / sum_k exp(e_ik)

Now the weights for node i sum to one. They are called attention coefficients.

Step 4: Weighted sum

The new vector is the weighted sum of projected neighbour vectors, followed by a non-linearity:

h_i_new = sigma( sum_j alpha_ij * z_j )

The self-loop ensures node i also looks at itself.

Step 5: Multiple heads

A single set of attention weights can be noisy. GAT runs K heads, each with its own W and a. In hidden layers, the K outputs are concatenated. In the last layer, they are averaged. Different heads can pick up different relationships, like asking several advisers with different tastes.

Step 6: Stack layers and train

Stacking two layers gives a two-hop view, as with GCN. Training is the same as for other node classifiers: cross-entropy on labelled nodes, dropout on features and on attention weights, and weight decay.

Why the original GAT is called static

In the original formula, the score splits into a part that depends on i and a part that depends on j, added before the non-linearity. Because LeakyReLU is monotonic, the ordering of neighbours by score is the same for every query node. GATv2 applies the non-linearity before the final projection, so the ranking can differ per query.

GAT   : e_ij = LeakyReLU( a_1^T z_i + a_2^T z_j )
GATv2 : e_ij = a^T LeakyReLU( W [ h_i || h_j ] )

Adding edge features

Edge attributes, such as the amount of a transfer or the type of a bond, can be added to the score by concatenating an edge vector, or by learning an embedding per edge type. This is common in relational and molecular work.

Shapes at a glance

For n nodes, input width f, H heads and per-head width d:

Why softmax over neighbours matters

The softmax runs over the neighbourhood of one node at a time. Two things follow. First, the weights always sum to one, so a node with ten neighbours and a node with two neighbours both produce a vector on a comparable scale. Second, the neighbours compete: raising one weight lowers the others. This competition is what lets the layer ignore weak links.

Attention dropout

GAT randomly drops attention weights during training, so each node sees a random subset of its neighbours in every step. This acts like data augmentation on the graph. It is one of the main reasons GAT can train on graphs with very few labels.

Variants you will meet

• GATv2, with dynamic attention.

• Edge-aware GAT, where edge vectors enter the score.

• Heterogeneous graph attention, where each node and edge type has its own parameters.

• Graph transformers, which let every node attend to every other node and inject graph structure through position encodings or biases.

• Sparse or top-k attention, which keeps only the strongest neighbours per node.

Key concepts and terms

A worked example

A numeric walk-through of attention weights

Node i has three neighbours A, B and C. Suppose the raw scores after the leaky ReLU are 1.0, 2.0 and 0.0.

1. Exponentiate: e^1.0 is about 2.72, e^2.0 is about 7.39, and e^0.0 is 1.0.

2. Sum: 2.72 + 7.39 + 1.0 = 11.11.

3. Weights: A is 0.245, B is 0.665 and C is 0.090.

4. If the projected vectors have a single value each, say 4, 8 and 2, then the new value is 0.2454 + 0.6658 + 0.090*2 = 0.98 + 5.32 + 0.18 = 6.48.

Neighbour B dominates because its score was highest. A fixed-average layer would have returned (4+8+2)/3 = 4.67. The attention layer gave more weight to the most relevant neighbour.

A single GAT layer in plain PyTorch

This sketch uses a dense adjacency for clarity. Real code uses sparse edge lists.

import torch
import torch.nn as nn
import torch.nn.functional as F

class GATLayer(nn.Module):
    def __init__(self, in_dim, out_dim, heads=4, concat=True, p=0.6):
        super().__init__()
        self.heads, self.out_dim, self.concat, self.p = heads, out_dim, concat, p
        self.W = nn.Linear(in_dim, heads * out_dim, bias=False)
        self.a_src = nn.Parameter(torch.empty(heads, out_dim))
        self.a_dst = nn.Parameter(torch.empty(heads, out_dim))
        nn.init.xavier_uniform_(self.a_src)
        nn.init.xavier_uniform_(self.a_dst)

    def forward(self, x, adj):
        n = x.size(0)
        z = self.W(x).view(n, self.heads, self.out_dim)       # (n, H, d)
        s = (z * self.a_src).sum(-1)                           # (n, H)
        d = (z * self.a_dst).sum(-1)                           # (n, H)
        e = F.leaky_relu(d[:, None, :] + s[None, :, :], 0.2)   # (n, n, H)
        mask = (adj + torch.eye(n)) > 0
        e = e.masked_fill(~mask[:, :, None], float("-inf"))
        alpha = F.softmax(e, dim=1)                            # over neighbours
        alpha = F.dropout(alpha, self.p, self.training)
        out = torch.einsum("ijh,jhd->ihd", alpha, z)           # (n, H, d)
        if self.concat:
            return F.elu(out.reshape(n, -1))
        return out.mean(dim=1)

Stacking two such layers, the first with concatenated heads and the second with averaged heads, reproduces the classic setup.

Using a library layer

import torch
from torch_geometric.nn import GATv2Conv

class Net(torch.nn.Module):
    def __init__(self, in_dim, hid, out_dim):
        super().__init__()
        self.c1 = GATv2Conv(in_dim, hid, heads=8, dropout=0.6)
        self.c2 = GATv2Conv(hid * 8, out_dim, heads=1, dropout=0.6)

    def forward(self, x, edge_index):
        x = torch.nn.functional.elu(self.c1(x, edge_index))
        return self.c2(x, edge_index)

Here edge_index is a two-row list of source and target node ids, the same edge list you would extract with a SQL query on a transactions table.

Inspecting what the model learned

After training, you can return the attention weights for a few nodes and list the highest-weighted neighbours. Do this with care. High attention shows what the layer used, not a proof of cause. Compare with a run where you remove the top neighbour and check whether the prediction changes.

Memory and speed

For a graph with E edges and H heads, one layer stores about E times H attention values. This is the main memory cost. A dense implementation like the one above stores n squared values per head and only suits tiny graphs. For large graphs, use sparse operations or neighbour sampling.

Building the edge list from a table

The query below uses placeholder table names to show how a transactions table becomes the edge list that a GAT layer expects.

SELECT
    sender_id   AS src,
    receiver_id AS dst,
    amount,
    created_at
FROM transfers
WHERE created_at >= DATE '2026-01-01';

The columns src and dst become the edge index. The amount can become an edge feature. The nodes are the accounts, and their columns form the node feature matrix.

Training loop outline

model = Net(in_dim=16, hid=8, out_dim=3)
opt = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)

for epoch in range(200):
    model.train()
    opt.zero_grad()
    out = model(x, edge_index)
    loss = torch.nn.functional.cross_entropy(out[train_idx], y[train_idx])
    loss.backward()
    opt.step()

    model.eval()
    with torch.no_grad():
        pred = model(x, edge_index).argmax(dim=1)
        acc = (pred[val_idx] == y[val_idx]).float().mean().item()

Keep the best validation accuracy and restore those weights at the end. Report test accuracy only once, at the end.

A second numeric check: self versus neighbours

Suppose node i has itself and two neighbours, with raw scores 0.5, 0.5 and 3.0. The weights are about 0.07, 0.07 and 0.86. The model has learned to trust one neighbour almost entirely. If this happens for most nodes, the graph is acting like a lookup from each node to a single partner. That can be right, for example in a one-to-one linking structure, or it can be a sign of overfitting. Compare with the validation score and with a version that has stronger attention dropout.

Strengths and limitations

Strengths

• Learns the importance of each neighbour, so noisy links can be down-weighted.

• Works in the inductive setting, since the layer needs only local neighbourhoods.

• Multi-head attention stabilises learning and captures several relations at once.

• Attention weights can give hints about which links drove a prediction.

• Handles neighbourhoods of varying size through softmax.

• Extends naturally to edge features and to relation types.

Limitations

• More compute and memory than GCN, because it scores every edge.

• Can overfit on small graphs with few labels, which is why heavy dropout is used.

• Over-smoothing still appears with depth.

• Attention weights are not guaranteed to be explanations.

• The original form uses static attention, which can limit expressiveness.

• On graphs where links carry little signal, attention cannot rescue the model.

• Large hubs create very wide softmax normalisations that may be hard to learn from.

How it compares with alternatives

A fair comparison also includes a model on flat features. In many industrial problems, a boosted tree on hand-built neighbour aggregates is the strongest simple baseline, and GAT must beat it to justify its cost.

Role in tabular foundation models

Tabular foundation models learn from raw tables with mixed column types, missing values and often no usable column names. GAT is not one of them, but it is closely related in two ways: it shares the attention mechanism that these models use, and it provides a way to bring relational context into a table.

Attention is the common mechanism

A tabular foundation model such as TabPFN or TabICL applies attention among cells, columns and rows. A GAT applies attention among linked nodes. In both, the model learns who should listen to whom. The difference is the mask. A graph supplies the mask in GAT. In a tabular model, the mask comes from the table layout: cells in the same row, cells in the same column, or labelled context rows versus query rows. See /model/architecture for how such masks are arranged.

A table as a graph of cells

You can view a table as a bipartite graph between rows and columns, where each cell is an edge with a value. A GAT-like model on this graph learns row vectors and column vectors from the cell values. This view handles mixed types if each edge carries an encoded value, and it handles missing values naturally, because a missing cell is simply a missing edge. It also needs no column names, since a column is just a node identified by its connections.

Relational tables

Rows in different tables are linked by keys. A GAT over this relational graph gives each row a vector that includes information from related rows, with learned weights. The vector can be added as extra input to a flat-table model. This is useful for customer, account and product data where one table does not hold the whole story.

Imputation

Missing values can be predicted from linked rows and from other columns. A graph attention model can fill a cell by attending to the cells and rows most related to it. This parallels masked cell modeling, where a model hides cells during pre-training and learns to restore them. For quality checks, see how imputation networks approach the task.

Anomaly detection

If attention weights concentrate on unexpected neighbours, or if a node's vector is far from what its neighbours predict, the node may be anomalous. Graph attention models are used for fraud rings, unusual device sharing and odd supply chain links. See anomaly detection for the general framing.

Semantic and sector classification

A table can be labelled by the domain it comes from, such as finance, health or logistics. Pooling attention-weighted column vectors gives a table-level embedding, which a head can classify. This is the idea behind semantic sector heads.

Pre-training and continual learning

A graph attention model can be pre-trained on many graphs, then adapted to a new one, much as tabular foundation models are pre-trained on large collections of synthetic or real tables and used with in-context examples. When data keeps changing, methods such as experience replay and EWC limit forgetting. For how these ideas are evaluated, see /model/benchmarks and /model/research.

Practical guidance and best practices

Data and graph preparation

• Check that edges exist at prediction time. Do not use links that are created after the label is known.

• Encode categories, scale numbers and add missing indicators before the first layer.

• Remove or flag duplicate edges.

• Keep a time-based split when the real task is forecasting.

Model choices

• Start with two layers and 4 to 8 heads of small width.

• Use concatenation for hidden layers and averaging for the output layer.

• Use heavy dropout on features and on attention.

• Try GATv2 when the original formulation underperforms.

• Add residual links for deeper stacks.

Training

• Use early stopping on validation nodes.

• Use a modest learning rate with weight decay.

• Run several seeds and compare with a GCN baseline.

• Watch for a large gap between training and validation loss, which signals overfitting.

Scaling

• Use neighbour sampling in the style of GraphSAGE for large graphs.

• Use sparse kernels, not dense matrices.

• Cap the neighbourhood size for hubs.

Interpretation

• Treat attention weights as hints.

• Test claims by removing edges and measuring the change in prediction.

• Report average attention by edge type, not only single examples.

A decision guide

Reproducibility

• Fix random seeds for model initialisation, dropout and data splits.

• Save the edge list and the split indices with the model.

• Log the number of heads, widths and dropout rates.

• Record library versions, since sparse kernels can change numeric results slightly.

Monitoring in production

• Track the distribution of attention weights. A sudden shift can signal a change in the graph.

• Track degree statistics. New hubs may appear after a data change.

• Re-evaluate on fresh labelled nodes on a schedule.

• Keep a fallback model that does not use the graph, so you can compare and switch if links break.

Common mistakes

• Using GAT on a graph where links are random or uninformative.

• Believing attention weights are proofs of cause.

• Forgetting to mask non-edges, which turns the layer into full attention.

• Setting too many heads on a tiny dataset and overfitting.

• Skipping self-loops, so the node ignores its own features.

• Leaking labels through features that were built from the target.

• Comparing GAT with a weak, untuned GCN and claiming a win.

• Ignoring memory growth with edges times heads on large graphs.

Frequently asked questions

What is a graph attention network?

A graph attention network is a graph neural network in which each node computes attention weights over its neighbours and takes a weighted sum of their features. The weights are learned, so the model decides which links matter. Several attention heads run in parallel to make learning stable.

How is GAT different from GCN?

GCN averages neighbours with fixed, degree-based weights. GAT learns the weights from the features of the node and its neighbour. This helps when neighbours are not equally useful, at the cost of more computation and more parameters.

What does multi-head attention do in GAT?

Each head has its own projection and scoring vector, so it can focus on a different pattern. In hidden layers the heads are concatenated, which widens the representation. In the output layer they are averaged. Using several heads also reduces the variance of training.

Is GAT inductive?

Yes, in the sense that its layer depends only on a node and its neighbours, not on the global graph structure. A trained model can be applied to new nodes or entirely new graphs, as long as the feature space is the same.

What is GATv2?

GATv2 is a later variant that changes the order of operations in the scoring function so that attention is dynamic, meaning the ranking of neighbours can depend on the query node. The original GAT produces a ranking that is the same for every query, which can limit what it can represent.

Are attention weights an explanation?

They are a useful hint about what the layer used, but they are not a guaranteed explanation. Different weights can lead to similar outputs, and later layers can change the meaning. To check importance, remove an edge or a neighbour and measure how the prediction changes.

Does GAT scale to large graphs?

It scales better than methods that need the full adjacency in dense form, but it still stores attention values for every edge and head. For very large graphs, use sparse implementations, neighbour sampling and mini-batches. Capping the neighbourhood size for hubs also helps.

Can GAT be used with tabular data?

Yes, when rows are linked by keys or when a table is turned into a graph of rows, columns and cells. It can provide relational context to a flat-table model. If rows are independent, a model built for flat tables is simpler and usually stronger.

How does attention in GAT relate to attention in transformers?

Both compute scores between pairs of items, turn them into weights with softmax and take weighted sums. A transformer lets every token attend to every other, while GAT restricts attention to graph neighbours. A transformer on a graph with a full mask is close to a graph attention layer on a complete graph.

When should I choose GAT?

Choose it when you suspect that neighbours differ in value, when you need to generalise to unseen nodes, or when edges carry features. Start with a simple baseline and move to GAT if the baseline leaves room for improvement and the cost is acceptable.

Glossary

• Attention coefficient: The normalised weight a node gives to a neighbour.

• Concatenation: Joining vectors end to end.

• Dynamic attention: Attention whose ranking depends on the query.

• Edge feature: A property attached to a link.

• Head: One independent attention computation.

• Inductive learning: Generalising to unseen nodes or graphs.

• Leaky ReLU: An activation that keeps a small slope for negative inputs.

• Masked attention: Attention restricted to allowed pairs.

• Message passing: Sending vectors along edges.

• Neighbourhood: The set of nodes linked to a node.

• Softmax: A function that turns scores into weights that sum to one.

• Static attention: Attention whose neighbour ranking does not depend on the query.

Summary

A GAT learns attention weights over each node's neighbours, then takes a weighted sum, using several heads for stability. It improves on the fixed averaging of GCN when neighbours differ in importance, and it works for unseen nodes like GraphSAGE. Its costs are more memory, a risk of overfitting on small graphs and the fact that attention is a hint, not a proof. For tabular foundation models, GAT matters because it shares the attention mechanism they use, offers a way to model tables as graphs of rows, columns and cells, and brings relational context into imputation, anomaly detection and table classification. See /model/architecture, /model/research and /model/benchmarks for further reading.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: