Glossary
What is FlashAttention? Definition and meaning
An attention kernel that computes exact attention in tiles to avoid reading and writing the large attention matrix to GPU memory.
Definition
FlashAttention: An attention kernel that computes exact attention in tiles to avoid reading and writing the large attention matrix to GPU memory.
In short: Fast, memory-efficient exact attention.
When it is used
Faster training and inference with long sequences.
Learn more
The full guide to FlashAttention explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: FinTech · Next term: Format-preserving encryption