GaLore

Overview

GaLore, short for gradient low-rank projection, is a memory-efficient training method. It lets you train all the weights of a model while keeping the optimizer states in a small low-rank space. For each large weight matrix, GaLore takes the gradient, projects it onto a low-rank subspace, runs the optimizer there, and projects the resulting update back to the full size. Because adaptive optimizers such as Adam store extra values for every parameter, shrinking that storage can cut a large share of training memory.

The key contrast with LoRA is what is low rank. LoRA restricts the weights, so the model can only move within a low-rank change. GaLore restricts the optimizer state and the gradient representation, while the weights themselves can move in any direction over the course of training, because the projection is refreshed from time to time. This makes GaLore useful both for pre-training and for full-parameter fine-tuning when memory for optimizer state is the bottleneck. It does not reduce activation memory, and it adds a periodic cost for computing projections.

What it is

GaLore is a training algorithm and optimizer wrapper. It does not change the model architecture and it does not add adapters. After training, you have an ordinary model with fully updated weights. There is nothing to merge.

The one-sentence version

Keep the full weights, but store the optimizer's running statistics for each matrix in a thin, projected form that is recomputed every so often from the gradient.

An everyday analogy

Imagine a manager tracking thousands of workers on a giant spreadsheet. Keeping a detailed history for every worker costs a lot of space. The manager notices that the day-to-day changes mostly move along a few main themes, such as shift patterns and seasonal load. So the manager records history only along those few themes, applies the change, and expands it back to every worker. Every worker still gets updated. The history record is just far smaller. Every so often the manager re-checks which themes matter most.

What is stored where

What GaLore is not

• It is not an adapter method. No extra parameters are added to the model.
• It does not make the number of trainable parameters smaller. All weights train.
• It does not remove activation memory, so long sequences still need checkpointing.
• It is not the same as compressing communication, although the idea of low-rank gradients appears there too.

Where it came from

The optimizer memory problem

Training with Adam keeps two extra values per parameter, the running mean and the running variance of gradients. In mixed precision these are typically held in 32-bit. For large models the optimizer state alone can be several times the size of the weights. This was a major barrier to training and full fine-tuning on limited hardware.

Earlier responses

• LoRA, 2021, avoided most of the cost by freezing the weights and training small matrices, but it limits updates to a low-rank change and can trail full training, especially in pre-training.
• 8-bit optimizers from the bitsandbytes library shrank state by quantizing it.
• Sharding methods such as DeepSpeed ZeRO and FSDP split state across devices.
• Memory-light optimizers that factor the second moment, such as Adafactor, reduced storage with structural approximations.
• Gradient checkpointing and mixed precision cut other memory terms.

The key observation

Studies of training dynamics noted that gradients of large layers tend to become low rank as training proceeds. In plain terms, the direction of useful change in a big matrix is concentrated in a few directions at any given time. GaLore builds on that observation. If the gradient is nearly low rank, you can keep optimizer state only for those directions.

The paper

The paper "GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection" appeared in 2024. It proposed projecting gradients onto a low-rank subspace chosen by a decomposition of the gradient, keeping the optimizer state in that subspace, and refreshing the subspace periodically. It reported large reductions in optimizer memory and demonstrated pre-training and fine-tuning of billion-parameter models on hardware with limited memory, with results close to full-rank training in its experiments.

After the paper

Follow-up work explored cheaper projections, quantized variants, and combinations with other optimizers. Implementations appeared in training libraries so that the method can be selected as an optimizer option.

How it works

Step 1: compute the gradient for a weight matrix

For a weight matrix W with shape m by n, backpropagation gives a gradient G of the same shape. This is the full-size gradient, and it exists for one layer at a time during the backward pass.

Step 2: choose a low-rank subspace

GaLore finds a projection matrix P from the gradient. A common way is to run a singular value decomposition of G and keep the top r left singular vectors (or right ones, depending on which side of the matrix is smaller). P has shape m by r, so it is thin. These r directions capture most of the energy of the gradient.

Step 3: project the gradient

Compute the small gradient R = P transposed times G. R has shape r by n. It is much smaller than G whenever r is much less than m.

Step 4: run the optimizer in the small space

Feed R to Adam (or another optimizer). The first and second moments are stored with shape r by n. They are the memory that is saved: instead of two matrices of size m by n, you store two of size r by n.

Step 5: project the update back

The optimizer outputs an update in the small space, call it N. Map it back with P times N, which has shape m by n, and apply it to the weights with the learning rate and a scale factor. Every entry of W can change, but the change lies in the span of P's columns for this step.

Step 6: refresh the subspace

Every T steps (for example, a few hundred), recompute P from the current gradient. Because the subspace changes over time, the weights are not confined to a fixed low-rank change. Over many refreshes, the accumulated updates can have high rank. This is the main reason GaLore can match full training better than a fixed LoRA update.

An analogy for the refresh

Think of exploring a city with a flashlight that lights up only a narrow strip. Each night you aim the flashlight along the street where you walked most. Over many nights you light up most of the city, even though on any one night the beam is narrow.

Step 7: scale

Because the update is a projection, a scale factor is applied so the update has a sensible size. This factor plays a role similar to the alpha in LoRA, and it is a hyperparameter.

The memory arithmetic

For one layer of size m by n with Adam:

• Full training stores 2 times m times n numbers for optimizer state.
• GaLore stores 2 times r times n numbers for state plus the projection matrix of m times r numbers.

If m equals n equals 4096 and r is 512, then full state is about 33.5 million numbers per layer. GaLore state is about 4.2 million plus a projection of about 2.1 million, around 6.3 million in total. That is roughly a five-fold reduction for that layer. Gains grow as the rank gets smaller relative to the layer size.

Per-layer updates

Some implementations apply the update layer by layer during the backward pass and free the gradient immediately. This avoids holding all gradients at once. Combined with 8-bit optimizer state, the approach has been used to fit very large models in modest memory. Per-layer updates change how gradient accumulation and clipping work, so they need care.

Cost of the decomposition

An SVD of a big matrix is not free. GaLore pays this only every T steps, which spreads the cost. Choosing T trades freshness of the subspace for overhead. A larger T is cheaper but may use a stale subspace.

Key concepts and terms

A worked example

A tiny numeric walk-through

Take a gradient G for a 3 by 2 weight matrix:

G = [ 2  0 ]
    [ 0  1 ]
    [ 0  0 ]

Rows correspond to output features and columns to input features. This gradient has rank 2, and the strongest direction is the first output feature, with singular value 2. Choose r = 1.

1. The top left singular vector is [1, 0, 0]. So P = [1, 0, 0] as a 3 by 1 matrix.
2. Project: R = P transposed times G = [2, 0], a 1 by 2 matrix.
3. Adam stores moments of size 1 by 2 instead of 3 by 2.
4. Suppose Adam's step on R is N = [0.1, 0]. Project back: P times N = [[0.1, 0], [0, 0], [0, 0]].
5. Apply to the weights, scaled by the learning rate.

Only the first output feature moved this step, because it carried the dominant gradient. At the next refresh, if the gradient has shifted toward the second output feature, P will point there, and that part of the weights will get updated. Over many steps all rows can learn.

A code example with a library optimizer

Several libraries expose GaLore as an optimizer. The names below are indicative. Check the documentation of your library version.

import torch
from transformers import AutoModelForCausalLM
# the optimizer class name may differ by library and version
from galore_torch import GaLoreAdamW

model = AutoModelForCausalLM.from_pretrained(
    "example-org/base-model", torch_dtype=torch.bfloat16  # placeholder
).cuda()

galore_params = []
regular_params = []
for name, module in model.named_modules():
    if isinstance(module, torch.nn.Linear) and any(
        key in name for key in ["attn", "mlp"]
    ):
        galore_params.append(module.weight)

galore_ids = {id(p) for p in galore_params}
regular_params = [p for p in model.parameters() if id(p) not in galore_ids]

param_groups = [
    {"params": regular_params},
    {
        "params": galore_params,
        "rank": 128,
        "update_proj_gap": 200,   # T, steps between projection refreshes
        "scale": 0.25,
        "proj_type": "std",
    },
]

optimizer = GaLoreAdamW(param_groups, lr=1e-5)

Only the large linear layers use the projection. Embeddings, norms, and biases use normal Adam state, since they are small or not matrix-shaped in the right way.

Training loop

for step, batch in enumerate(loader):
    loss = model(**batch).loss
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    optimizer.zero_grad(set_to_none=True)
    scheduler.step()

A from-scratch sketch of one projected step

This sketch shows the idea for a single matrix and a simple momentum update. It is not a full optimizer.

import torch

def galore_step(W, G, state, r=64, T=200, lr=1e-3, beta=0.9, scale=0.25, step=0):
    if step % T == 0 or "P" not in state:
        U, S, Vh = torch.linalg.svd(G.float(), full_matrices=False)
        state["P"] = U[:, :r]                      # m x r
        state["M"] = torch.zeros(r, G.shape[1], device=G.device)
    P = state["P"]
    R = P.T @ G.float()                            # r x n
    state["M"] = beta * state["M"] + (1 - beta) * R
    update = P @ state["M"]                        # m x n
    W.data.add_(update.to(W.dtype), alpha=-lr * scale)
    return state

When the subspace is refreshed, real implementations decide how to treat the old moments. A simple choice is to reset them, as here. Other choices exist.

Memory estimate for a real-size layer

The saving grows as r shrinks, while quality usually declines if r is too small. The right rank is a trade-off you find by experiment.

Strengths and limitations

Strengths

• Trains all weights, so the final model is a normal full-parameter model.
• Large cut in optimizer state memory.
• Suitable for pre-training as well as fine-tuning, where LoRA is often weaker.
• No adapters, nothing to merge, no inference cost.
• The refreshed subspace lets accumulated updates reach high rank over time.
• Works with 8-bit optimizers and with per-layer updates for even lower memory.
• Compatible with data parallel and sharded training in principle.

Limitations

• Does not reduce activation memory.
• Gradients still need to exist in full size for a layer, at least briefly.
• Periodic SVD adds compute and some complexity.
• Extra hyperparameters: rank, update interval, and scale.
• The low-rank assumption may not hold equally well for every layer or every phase of training.
• Resetting or transforming moments at refresh time can disturb optimization if done carelessly.
• Fine-tuning many tasks leaves you with many full model copies, unlike adapter methods that save small files.
• Multi-GPU integration with sharding methods needs care and varies by library.
• The saving applies mainly to large matrix layers.

When GaLore is a good fit

• Optimizer memory is the main bottleneck for full-parameter training.
• You want a full-weight result, for example in continued pre-training.
• You can afford the storage of one full model per task.

When another method fits better

• You need many small task files. Use LoRA or DoRA.
• The base model does not even fit for forward and backward. Use QLoRA.
• You can shard state across several devices. Use DeepSpeed ZeRO or FSDP.

How it compares with alternatives

GaLore versus LoRA in plain words

• LoRA says: only a low-rank change is allowed. GaLore says: any change is allowed, but we keep the bookkeeping in a low-rank space.
• LoRA gives small files. GaLore gives a full model.
• LoRA is excellent for many tasks on one base. GaLore is better when the goal is to change the whole model.
• They are not strictly rivals. Some work combines low-rank ideas in both places.

GaLore versus sharding

Sharding spreads memory across devices but needs several devices and communication. GaLore reduces the memory itself and can help on a single device. On a cluster, the two can be complementary, with the caveat that integration is not always trivial.

GaLore versus 8-bit optimizers

Both attack optimizer state size. 8-bit methods shrink each number. GaLore shrinks the number of stored values. Combining them gives a larger saving, with the usual risk that approximation errors add up.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns, mixed numeric, categorical, date, and text values, missing cells, and often no column names. They are pre-trained on large collections of synthetic or real tables, solve tasks through in-context learning, and can be updated continually. See /model/architecture, /model/research, and /model/benchmarks. GaLore is relevant mostly to training and updating these models, not to using them.

Pre-training on tables with limited memory

Pre-training a table model means running many steps over huge numbers of generated or collected tables. Adapter methods do not suit pre-training well, because pre-training needs to change the whole network. GaLore keeps full-parameter learning while shrinking optimizer state. This lets a model with billions of parameters be pre-trained on fewer or smaller accelerators than a plain Adam setup would need.

Why attention-heavy table models have large matrices

Models such as TabPFN, TabICL, and TabSTAR use attention along rows and along columns, and projection layers for each cell. Large linear layers in these blocks dominate parameter count at scale, and they are the layers where GaLore saves most. Small components such as value encoders, type embeddings, and missing-value indicators are better left on normal optimizer state.

Continued pre-training and full updates

When a table model must absorb a new domain, an adapter may not be enough, for example when the new tables follow structure the model has never seen. Continued pre-training updates all weights on new data. GaLore makes this affordable when optimizer memory would otherwise block it. The result is a full model that already contains the new knowledge, with nothing to merge.

Continual learning

GaLore does not prevent forgetting by itself, because all weights change. Combine it with replay of old tables, a small learning rate, or regularization such as EWC. In contrast, adapter methods protect the base by freezing it. A practical plan is to use adapters for routine per-customer tuning and GaLore for scheduled refreshes of the shared base.

Mixed types, missing values, and no column names

A full update lets the model change how it embeds each cell type and missing indicator everywhere in the network. If a new corpus has a different missingness mechanism or numeric scale, full updates can learn it in the earliest layers as well as the deepest. With no column names, the model depends on value statistics, and a full update can reshape these internal statistics broadly.

In-context learning

Pre-training with GaLore targets the in-context learning ability itself. The model sees a context of labeled rows and a query row, and learns to predict the label. Because pre-training changes all weights, optimizer memory is a real limit for large context sizes and wide models. GaLore reduces one of the memory terms. Activation memory for long contexts is still the main cost, so combine with checkpointing.

Imputation, anomaly detection, and sector classification

• Imputation: Pre-training with a masked-cell objective is a full-parameter task. See masked cell modeling and imputation networks.
• Anomaly detection: A shared base trained on many normal behaviors can be refreshed with full updates, then specialized with adapters. See anomaly detection.
• Sector and semantic classification: Training the encoder and the classification heads together on a broad taxonomy is a full-parameter job. See semantic sector heads.

Where it sits in a training plan

Honest caveats

• Many tabular models are small. For a model with tens or hundreds of millions of parameters, normal Adam memory is usually fine and GaLore adds complexity for little gain.
• The low-rank gradient assumption was studied mostly in language models. Check it on your table model by looking at the singular values of gradients.
• Always compare against a full-rank baseline on held-out tables.

Practical guidance and best practices

Choosing the rank

• Start with a rank around one quarter to one half of the layer width for pre-training, and lower for fine-tuning.
• Lower rank saves more memory but may slow convergence. Increase the rank if loss lags behind a full-rank reference.
• Evaluate the trade-off with a short pilot run.

Choosing the update interval

• Values in the low hundreds of steps are a common starting point.
• Shorter intervals track the gradient better but cost more compute.
• Longer intervals are cheaper but may use a stale subspace and slow learning.

Choosing the scale and learning rate

• Treat the scale as a tunable constant, often below one.
• Keep a warmup and a decaying schedule.
• Because the update is projected, the learning rate that works for full Adam may need adjustment.

Which layers to apply it to

• Apply to large linear layers in attention and feed-forward blocks.
• Leave embeddings, norms, biases, and small heads on a regular optimizer.
• Check shapes: the projection works on matrices, so skip one-dimensional parameters.

Combining with other memory tools

• Use bf16 or fp16 mixed precision. See mixed precision.
• Use gradient checkpointing to cut activation memory.
• Consider 8-bit optimizer state from bitsandbytes.
• Use per-layer updates if your library supports them, and check how they interact with accumulation and clipping.
• In multi-GPU setups, verify support with FSDP or DeepSpeed ZeRO in your library version.

A pilot plan before a long run

1. Pick a small model of the same family and a short data slice.
2. Train a full-rank Adam baseline and record loss and memory.
3. Train GaLore at two or three ranks with the same seed.
4. Compare loss at equal steps and peak memory.
5. Choose the smallest rank whose loss is close to the baseline.
6. Only then launch the large run, with the same interval and scale.

Monitoring

• Track the training loss against a full-rank run on a small model.
• Log the energy captured by the top r singular values of the gradient. If it is low, your rank may be too small.
• Log time per step, since SVD refresh adds spikes.

Evaluation

• Evaluate the final model as a normal model. There is nothing to merge.
• Compare to a LoRA run of equal memory budget.
• Check for forgetting on earlier tasks if you use it for continual updates.

Troubleshooting table

Storage and versioning

• Each GaLore run produces a full checkpoint. Plan for storage and use delta tools or compression if you keep many.
• Record rank, interval, scale, and library version in the run metadata.

Common mistakes

• Expecting GaLore to reduce activation memory. It does not.
• Applying it to every parameter, including embeddings and biases.
• Setting the rank too small and blaming the method for poor convergence.
• Refreshing the subspace too often and wasting compute, or too rarely and getting stale updates.
• Ignoring how moments are handled at a refresh.
• Using the full-Adam learning rate without checking it.
• Assuming it replaces sharding when the model is too big for even one forward pass.
• Choosing it for per-task tuning where small adapter files would serve better.
• Forgetting that all weights change, so earlier skills can be forgotten.
• Not comparing to a baseline with 8-bit optimizer state, which may be simpler.
• Running per-layer updates together with gradient accumulation or clipping without checking that the combination is supported.
• Treating published memory figures as guarantees. Real savings depend on model shape, rank, precision, and library.

Frequently asked questions

What is GaLore?

GaLore is gradient low-rank projection, a training method that keeps the optimizer state for each large weight matrix in a small low-rank space while still updating all weights. It reduces optimizer memory so that full-parameter training and fine-tuning fit on smaller hardware.

How is GaLore different from LoRA?

LoRA freezes the weights and trains a small low-rank add-on, so the model change is low rank. GaLore trains all weights and makes the gradient and optimizer state low rank. LoRA yields small adapter files. GaLore yields a full model.

Does GaLore reduce activation memory?

No. Activations needed for backpropagation are the same as in normal training. To reduce them, use gradient checkpointing, shorter sequences, or smaller batches.

Why does a low-rank projection of the gradient work?

Studies of training dynamics suggest that gradients of large layers often concentrate in a few directions. If the gradient is nearly low rank, storing optimizer statistics for those directions captures most of the useful signal. Refreshing the directions over time lets the weights move in many directions overall.

What are the main hyperparameters?

The rank of the projection, the number of steps between subspace refreshes, and a scale factor for the update. Together with the usual learning rate and schedule, these control the trade-off between memory, speed, and quality.

Can GaLore be used for pre-training?

Yes. This is one of its main uses, since adapter methods are not designed to change the whole model. The original work demonstrated pre-training with reduced optimizer memory. Results depend on the model and the setup.

Can I combine GaLore with quantization?

Combinations with 8-bit optimizer state and with lower precision weights have been explored. Each approximation adds some error, so test the combination against a simpler baseline before committing to a long run.

Is GaLore useful for small models?

Usually not much. If the optimizer state fits comfortably in memory, normal Adam is simpler and avoids the cost of computing projections. The benefit grows with model size.

Does GaLore help with catastrophic forgetting?

Not by itself, because every weight can change. If forgetting matters, add replay of earlier data, a lower learning rate, or regularization. Adapter methods that freeze the base are safer for routine multi-task use.

How does GaLore relate to tabular foundation models?

It is a tool for training and refreshing them. It helps when a large table model needs full-parameter pre-training or continued pre-training and optimizer memory is the limit. For cheap per-schema tuning, adapter methods are usually the better fit.

Glossary

• Activation memory: Memory for intermediate results kept for the backward pass.
• Adam: An optimizer that keeps running averages of gradients and squared gradients.
• Gradient: The signal that says how to change a weight to reduce the loss.
• Low-rank: Described by a small number of independent directions.
• Moment: A running average kept by an optimizer.
• Optimizer state: The extra values an optimizer stores per parameter.
• Projection: Mapping a matrix into a smaller space and back.
• Rank: The number of directions kept.
• Singular value decomposition: A factorization into main directions and their strengths.
• Subspace: The set of directions spanned by the projection columns.
• Update interval: The number of steps between recomputations of the projection.
• Full-parameter training: Training every weight in the model.

Summary

GaLore trains all the weights of a model while storing optimizer state in a low-rank space. For each large matrix it projects the gradient onto the top directions, runs the optimizer there, projects the update back, and refreshes the directions every so often. This cuts the optimizer memory that makes full training expensive, without adapters and without any merge step. It does not reduce activation memory, adds hyperparameters and periodic decomposition cost, and yields full model checkpoints.

For tabular foundation models, GaLore is most useful for pre-training and for scheduled full refreshes of large table models, where adapter methods are too limited. Per-schema and per-customer tuning is still better served by LoRA, DoRA, rsLoRA, and QLoRA. Related reading: full fine-tuning, DeepSpeed ZeRO, FSDP, and the architecture and benchmarks pages.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: