Glossary
What is GaLore? Definition and meaning
Projects gradients into a low-rank space so optimizer states take far less memory, while training all weights.
Definition
GaLore: Projects gradients into a low-rank space so optimizer states take far less memory, while training all weights.
In short: Memory-efficient training by projecting gradients.
When it is used
Pre-training or full fine-tuning when optimizer memory is the bottleneck.
Learn more
The full guide to GaLore explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Function calling · Next term: GAN