Federated learning

Overview

Federated learning trains one shared model across many data holders without moving their raw data to a central place. Each participant trains on its own data and sends back only model updates. A coordinator combines the updates into a better shared model and sends it out again. The data stays where it was created.

This matters when data is private, regulated, too large to move, or owned by parties who do not trust each other. Banks, hospitals, factories, and phone makers all face this. For tabular foundation models, federated learning offers a way to learn from many separate tables, each of which cannot be pooled, and to adapt a general model to local data without exposing it.

What it is

Federated learning, often shortened to FL, is a form of distributed machine learning with a twist. In ordinary distributed training, data is shuffled across machines inside one organization's cluster. In FL, the data lives with different owners, and it must not leave.

Picture a group of chefs who each have a private family recipe book. They want a better shared recipe, but none will hand over the book. Instead each chef tries a recipe at home and reports only a note such as "add a little more salt". A coordinator collects the notes, merges them, and shares the improved recipe. After many rounds the shared recipe is good, and no book was ever copied.

Two main settings

• Cross-device FL: a very large number of small, unreliable clients, such as phones. Each holds little data and may drop out at any moment.
• Cross-silo FL: a small number of reliable, large clients, such as hospitals or banks. Each holds a lot of data and is usually online.

Tabular work most often falls into the cross-silo setting, since tables belong to organizations.

How the data is split

Horizontal FL is the easiest and the most common. Vertical FL needs a way to match rows across parties without revealing identities, which adds cryptographic work.

What it is not

• It is not automatically private. Model updates can leak information unless extra protections are added.
• It is not a replacement for good data. Poor local data gives poor updates.
• It is not the same as transfer learning, although the two are often combined.

Where it came from

Early roots lie in distributed optimization and in privacy research in the 2000s. Work on secure multi-party computation and homomorphic encryption in earlier decades supplied tools for computing on data without seeing it. Differential privacy, formalized in the mid 2000s, gave a rigorous way to bound what any output reveals about one person.

The term federated learning was popularized by a 2016 paper on communication-efficient learning of deep networks from decentralized data. It introduced the Federated Averaging algorithm, known as FedAvg, and applied it to keyboard prediction on phones. Work on secure aggregation, which lets a server see only the sum of client updates, followed in 2017.

From 2018 onward, research expanded quickly. Topics included handling non-identically distributed data, personalizing models for each client, reducing communication, defending against poisoned updates, and building open frameworks. Cross-silo uses grew in health care, finance, and industry, where data sharing rules are strict. Regulations that limit data movement, such as those on personal data in many regions, pushed interest higher.

More recently, work has looked at federated fine-tuning of large pre-trained models. The idea is to start from a strong general model and adapt it locally, sending only small updates. The research overview places this alongside table-focused foundation models.

How it works

The standard process is a loop of rounds. The description below follows FedAvg, the baseline.

Step 1: The server starts a global model

The coordinator creates or loads an initial model. It might be randomly initialized, or it might be a pre-trained model, which is usually better.

Step 2: The server selects clients

In each round, the server picks a subset of clients. In cross-device settings it samples a small fraction. In cross-silo settings it often uses all participants.

Step 3: Clients receive the current model

Each chosen client downloads the current global weights. Only weights travel, not data.

Step 4: Clients train locally

Each client runs a few steps or epochs of training on its own data, starting from the global weights. This is ordinary supervised learning or another learning type, done privately.

Step 5: Clients send back updates

Each client sends back its new weights or the difference from the weights it received. It may also send the number of examples it used.

Step 6: The server aggregates

FedAvg takes a weighted average of the client updates, with weights proportional to the number of local examples. A client with twice the data has twice the influence. The result becomes the new global model.

Step 7: Repeat

The loop continues for many rounds until the global model stops improving or a budget is reached.

The FedAvg update in a line

The new global weights equal the sum, over clients, of each client's share of the total data times that client's locally trained weights. This is a simple weighted mean.

Why it can work

Local training moves weights in useful directions for each client. Averaging combines these moves, like merging the opinions of several experts who each studied a different slice of the world. If the slices are similar, the average is a good compromise.

Why it is hard

• Client data is rarely identical in distribution. This is called non-IID data.
• Clients differ in size, speed, and availability.
• Communication is slow or costly compared with local computing.
• Updates can reveal private information.
• Some clients may be faulty or malicious.

Privacy protections

Secure aggregation

Clients mask their updates with random values that cancel out when summed. The server learns only the total, not any single client's update. It is like a group of people each writing a number on a card, adding a secret random offset, and only the sum of all cards being made public.

Differential privacy

Each client clips its update to a maximum size and adds calibrated noise. This limits how much any one record can change the output. It brings a trade-off: more privacy means more noise and lower accuracy. The trade-off is central to any private deployment.

Homomorphic encryption and trusted hardware

Encrypted updates can be summed without decrypting. Trusted execution environments offer another route. Both add cost and complexity.

Why raw data staying local is not enough

Studies have shown that gradients and weights can sometimes be used to reconstruct training examples or to infer whether a record was in the training set. That is why serious deployments add secure aggregation, noise, or both.

Handling non-IID data

• Personalization: keep a shared base and let each client fine-tune a small part.
• Regularized local training: add a term that keeps local weights close to the global model.
• Clustered FL: group similar clients and train a model per group.
• Server momentum or adaptive optimizers: smooth noisy updates.
• Data sharing of tiny public sets: reduce drift, when allowed.

Tree models in FL

Neural networks are not the only choice. Gradient boosted trees can be trained across parties by sharing encrypted or summed split statistics, such as gradient histograms, instead of weights. This matters a lot for tables, since tree ensembles such as XGBoost are strong baselines.

Communication efficiency

Sending a large model every round is expensive. Several tricks reduce the load.

• Run more local steps per round, so fewer rounds are needed.
• Send only the change in weights, not the full weights.
• Quantize updates to fewer bits.
• Sparsify updates by sending only the largest changes.
• Update only a small part of the model, such as adapter layers.
• Start from a pre-trained model so fewer rounds are needed.

Robust aggregation

A plain average is easy to corrupt. A single client sending huge values can pull the result far off. Robust rules limit this damage.

• Clip each update to a maximum norm.
• Use the coordinate-wise median instead of the mean.
• Use a trimmed mean that drops the highest and lowest values.
• Down-weight clients whose updates differ strongly from the rest.

Each rule gives up a little accuracy on clean data to gain safety against bad data.

Key concepts and terms

Privacy budget

In differential privacy, a number often written as epsilon measures how much privacy is spent. A smaller value means stronger privacy. Every extra round or query uses some of the budget, so total training length matters. Teams choose a budget up front and plan the number of rounds to fit it.

A worked example

The sketch below simulates FedAvg for a logistic regression on three clients. It does not use a network. It shows the logic of local training and weighted averaging with plain NumPy.

import numpy as np

rng = np.random.default_rng(0)

def make_client(n, shift):
    X = rng.normal(shift, 1.0, size=(n, 4))
    true_w = np.array([1.5, -2.0, 0.5, 1.0])
    p = 1 / (1 + np.exp(-(X @ true_w)))
    y = (rng.random(n) < p).astype(float)
    return X, y

clients = [make_client(400, 0.0), make_client(150, 0.5), make_client(900, -0.3)]

def local_train(w, X, y, lr=0.1, steps=20):
    w = w.copy()
    for _ in range(steps):
        p = 1 / (1 + np.exp(-(X @ w)))
        grad = X.T @ (p - y) / len(y)
        w -= lr * grad
    return w

def accuracy(w, X, y):
    return (((X @ w) > 0).astype(float) == y).mean()

w_global = np.zeros(4)
sizes = np.array([len(c[1]) for c in clients], dtype=float)

for rnd in range(30):
    local_ws = [local_train(w_global, X, y) for X, y in clients]
    w_global = sum((n / sizes.sum()) * w for n, w in zip(sizes, local_ws))

for i, (X, y) in enumerate(clients):
    print("client", i, "accuracy", round(accuracy(w_global, X, y), 3))
print("global weights:", np.round(w_global, 2))

Reading the example

• Each client has a different amount of data and a slightly different feature shift. That mimics non-IID data.
• Local training starts from the current global weights and runs 20 gradient steps.
• Aggregation is a mean weighted by client size, which is FedAvg.
• No client ever shares X or y. Only weight vectors move.
• The global weights should approach the generating weights, and accuracy should be similar across clients.

A numeric walk-through of one aggregation

Three clients have 400, 150, and 900 rows, so the total is 1,450. The weights are about 0.276, 0.103, and 0.621. Suppose the first weight coordinate comes back as 1.2, 1.8, and 1.4. The new global value is 0.276 times 1.2 plus 0.103 times 1.8 plus 0.621 times 1.4. That is 0.331 plus 0.186 plus 0.869, or about 1.39. The largest client pulls the average toward its own value. This is fair in terms of data size, but a small client with a very different distribution may be under-served, which is why personalization can matter.

Adding noise for privacy

A simple extension clips each client update to a norm of at most C and adds Gaussian noise to the average. The clipping stops any one client from dominating. The noise hides individual contributions. Both steps cost accuracy, so you tune C and the noise level against the privacy target.

A vertical example in words

A bank holds income and credit history for 100,000 people. A retailer holds purchase history for many of the same people. Neither will share its columns. To train a credit model with both views, the parties first find the overlapping people using private set intersection, which reveals only the shared IDs or even nothing beyond the match count. Then each party computes partial results on its own columns and exchanges encrypted intermediate values. The final model uses both sets of columns, yet neither party sees the other's raw fields.

Settings and typical sizes

Applications

• Health care: hospitals train a shared risk model without moving patient records.
• Finance: banks train fraud and credit models across institutions.
• Manufacturing: plants learn failure patterns without sharing process secrets.
• Telecom and devices: next-word and usage prediction learned on the device.
• Retail: stores or partners learn demand patterns jointly.
• Energy and utilities: sites learn load forecasts while keeping meter data local.
• Public sector: agencies learn from data that law keeps in separate systems.

A short story of a cross-silo project

Five regional clinics want a model that predicts missed appointments. Each has a few thousand rows. Alone, none gets a good model. They agree on a common set of eleven columns, such as age band, lead time, and prior misses. A neutral coordinator runs FedAvg with secure aggregation. After twenty rounds, the shared model beats every local-only model on each clinic's own validation data. One clinic with unusual patients gains less, so it adds a short local fine-tuning step. Nobody ever moved a patient record.

Strengths and limitations

Strengths

• Raw data stays with its owner, which eases legal and trust concerns.
• It lets organizations learn together when they could not pool data.
• It cuts the need for large central data stores and transfers.
• It can reach data that is otherwise unreachable, such as data on devices.
• It can give a model more diverse training data than any one party holds.
• It pairs well with pre-trained models, since only small updates need to move.

Limitations

• It is not private by itself. Extra tools add cost and reduce accuracy.
• Non-IID data slows convergence and can hurt quality.
• Communication can dominate cost, especially for large models.
• Debugging is harder because the data cannot be inspected centrally.
• Clients may differ in schema, which is a real problem for tables.
• Evaluation is tricky, since there is no central test set.
• Malicious clients can poison the model, so defenses are needed.
• Operations are complex: scheduling, versioning, and failures across parties.

When it is a good fit

How it compares with alternatives

Federated learning versus synthetic data

Both aim to share learning without sharing raw records. Federated learning shares model updates. Synthetic data sharing releases generated rows that mimic the real ones. Synthetic data is easy to hand to anyone but may leak information if the generator memorized records. Federated learning keeps data in place but needs a live training setup. Some teams combine them: each party trains a local generator, and only generator updates are shared.

Federated learning versus other learning types

• It is a training setting, not a learning goal. You can run supervised, unsupervised, self-supervised, or reinforcement learning in a federated way.
• It complements meta-learning, which can help a global model adapt quickly to each client.
• It complements few-shot learning, which helps small clients benefit from the shared model.

Role in tabular foundation models

Tabular foundation models learn from raw tables. Most valuable tables are private. Federated ideas link these two facts.

The data access problem

The best training data for table models, such as customer, transaction, patient, and sensor tables, sits inside organizations that cannot share it. Federated learning offers a path to learn from many such tables without centralizing them. The model improves, and each owner keeps control.

Pre-training versus adaptation

Two uses are common.

• Federated pre-training: many parties help train the general model. This is costly and rare, because the model is large and tables have different schemas.
• Federated or local adaptation: a pre-trained model is given to each party, which adapts or uses it in context on private data. Only small updates, or none at all, leave the site.

For table models that support in-context learning, the second route is very attractive. A party can put its own labeled rows into the model's context at prediction time, and nothing is trained or sent. The private data never leaves the local machine. See few-shot learning and TabPFN for how a small number of rows can drive predictions.

Row and column structure

Horizontal FL fits tables well: each party holds rows, and all share a column set. Vertical FL splits columns across parties and needs private matching of rows. A model with attention across rows and columns, as described on the architecture page and built on transformers, does not depend on a fixed column order. That helps when parties have slightly different column sets.

Mixed types, missing values, and schema differences

Parties rarely use identical schemas. One may store age as a number and another as a band. One may have many missing fields. Because a foundation model learns a shared way to handle numbers, categories, dates, and missing cells, it tolerates this variation better than a model trained for one schema. Local imputation, as in imputation networks, can be done on each site, so raw rows never move.

No column names

If shared tables use anonymized or cryptic headers, the model still has to work. A model that infers column roles from values and relations can be used across sites without agreeing on names. When names exist, semantic matching can help align columns across parties, which supports sector and semantic classification as well.

Synthetic pre-training reduces what must be shared

Table models such as TabPFN are pre-trained on generated tables, built with synthetic data generation. Since the pre-training data is synthetic, no private data is needed for the base model. Federated learning then only has to handle a light adaptation step, if any. This is a clean separation: public synthetic data for general skill, private data for local fit.

Continual learning across rounds

FL is naturally continual. Each round brings new local data. A federated table model must absorb new information without forgetting old patterns, and it must cope with drift at individual sites. Techniques from continual learning, such as replay and regularization, carry over.

Anomaly detection and fraud

Fraud and failure patterns are rare at any single site, but common across many. A federated detector can learn from them all. A table model can score how unusual a row is compared with the learned pattern, which supports anomaly detection, while each site keeps its transactions private.

Evaluation

Scores for federated table models should be reported per client and in aggregate, since averages can hide a client the model serves poorly. General approaches to scoring are covered on the benchmarks page.

Practical guidance and best practices

Decide if FL is needed

• Can the data be pooled legally and safely? If yes, central training is simpler.
• Would a pre-trained model used locally already solve the problem? If yes, no training round may be needed.
• Are the schemas similar enough? If not, plan a mapping step.
• Is there a trusted coordinator, or do you need cryptographic protection?

Design the setup

• Pick the partition type: horizontal, vertical, or hybrid.
• Align features and definitions across parties before training starts.
• Choose cross-silo for institutions and cross-device for many small clients.
• Start from a strong pre-trained model if possible. It cuts the number of rounds.
• Decide on privacy tools up front: secure aggregation, differential privacy, or both.

Schema alignment checklist

• Agree on the list of columns and their meaning.
• Agree on units, such as dollars versus cents, or Celsius versus Fahrenheit.
• Agree on category codes and how to treat unknown values.
• Agree on how to represent missing values.
• Agree on the target definition and the time window.
• Run a dry test on fake data from each site to catch mismatches early.

Train well

• Use a moderate number of local steps. Too many cause client drift.
• Weight clients by data size, but check that small clients are not ignored.
• Use adaptive server optimizers for noisy updates.
• Compress updates to cut bandwidth.
• Handle stragglers by timeouts or asynchronous updates.
• Monitor per-client metrics every few rounds.

Evaluate fairly

• Keep a local validation set at each client.
• Report the mean and the worst client.
• Compare with local-only training and with a pooled baseline if one is available.
• Test robustness against a faulty or malicious client.

Govern and document

• Record who participates, what data fields are used, and which privacy settings apply.
• Version the global model and the client code.
• Agree on rules for withdrawing a participant.
• Audit for leakage with membership inference tests.

A short checklist

• Is the partition type clear?
• Are schemas aligned?
• Is the privacy method chosen and tested?
• Is there a local-only baseline to beat?
• Are per-client results visible?
• Is there a plan for failures and dropouts?

Common mistakes

• Assuming FL is private by default and skipping secure aggregation or noise.
• Ignoring schema differences, so the same column name means different things.
• Training too many local steps on skewed data, which causes drift.
• Looking only at average accuracy and missing a client with poor results.
• Comparing against nothing. Without a local-only baseline, you cannot tell if FL helped.
• Using FL when the data could be safely pooled, adding cost for no gain.
• Sending full large models every round without compression.
• Forgetting poisoning risk. One bad client can damage the shared model.
• Spending the privacy budget too fast by running too many rounds.
• Neglecting operational basics such as versioning, logging, and failure handling.

Frequently asked questions

What is federated learning in simple words?

Federated learning trains a shared model across many data owners without gathering their data in one place. Each owner trains on its own data and sends back only model updates. A coordinator merges the updates, and the process repeats until the model is good.

Is federated learning really private?

It is more private than pooling data, but not private by default. Model updates can leak details about training records. Real deployments add secure aggregation, differential privacy noise, or encryption, and they test for leakage.

What is FedAvg?

FedAvg, short for Federated Averaging, is the basic algorithm. Clients train locally for a few steps, then the server takes a weighted average of their models, with weights based on how much data each client has. It is simple and works well when client data is similar.

What is non-IID data and why does it matter?

Non-IID means client data is not drawn from the same distribution. One bank may serve young customers and another older ones. Such differences make local models pull in different directions, which slows learning and can lower quality for some clients.

What is the difference between cross-device and cross-silo?

Cross-device FL involves a huge number of small, unreliable clients like phones. Cross-silo FL involves a few reliable, large clients like hospitals or banks. Tabular business data usually falls into the cross-silo kind.

Can federated learning work with tabular data?

Yes. Horizontal setups, where parties share columns but hold different rows, are common for tables. Tree-based methods can use shared statistics instead of weights, and neural table models can use weight averaging. Schema alignment is the main practical challenge.

How is federated learning different from distributed training?

Distributed training spreads one organization's data across machines to speed up learning, and the data can be shuffled freely. Federated learning keeps data with separate owners who do not share it, so it faces non-IID data, limited communication, and privacy needs.

How does federated learning relate to tabular foundation models?

Table foundation models need many diverse tables, but valuable tables are private. Federated learning can adapt such models to local data without moving it, and in-context learning can avoid training altogether. Models pre-trained on synthetic tables also reduce the need to share real data for the base model.

Does federated learning reduce accuracy?

It can. Non-IID data, privacy noise, and limited rounds may lower accuracy compared with pooled training. Good design, such as a strong starting model, personalization, and careful tuning, often closes much of the gap. It still beats training only on one party's small data in many cases.

What are the main risks?

The main risks are information leakage from updates, poisoning by a bad client, uneven benefit across clients, and operational failures. Each has known mitigations: secure aggregation and noise for leakage, robust aggregation for poisoning, personalization for fairness, and solid engineering for operations.

Glossary

• Aggregation: Combining client updates into a global model.
• Client: A participant that holds data and trains locally.
• Client drift: Local models moving away from a good global solution.
• Coordinator: The server that orchestrates rounds and aggregates updates.
• Cross-device: FL with many small clients.
• Cross-silo: FL with few large clients.
• Differential privacy: A formal limit on how much an output reveals about one record.
• FedAvg: Federated Averaging, a size-weighted mean of client models.
• Gradient: The direction in which the loss changes fastest.
• Homomorphic encryption: Encryption that allows computing on encrypted values.
• Horizontal FL: Clients share features but hold different rows.
• Membership inference: An attack that tests whether a record was in the training set.
• Non-IID: Not independent and identically distributed across clients.
• Personalization: Tailoring the global model to each client.
• Poisoning: Sending harmful updates to damage the model.
• Round: One cycle of distribute, train, return, aggregate.
• Secure aggregation: A protocol that reveals only the sum of client updates.
• Straggler: A slow client that delays a round.
• Vertical FL: Clients share rows but hold different features.

Summary

Federated learning lets many parties train one shared model while keeping their data in place. The standard method, FedAvg, alternates local training with weighted averaging. It is used in cross-device and cross-silo forms, with horizontal, vertical, or hybrid splits of data.

It is not private on its own. Real use needs secure aggregation, differential privacy, or both, along with checks for leakage, poisoning, and uneven benefit. Non-IID data and schema differences are the main practical hurdles, and good evaluation reports results per client.

For tabular foundation models, federated ideas matter because valuable tables are private. Models pre-trained on synthetic tables need no private data for the base, and in-context learning can adapt them on site without training or sending anything. Where updates are shared, they support imputation, anomaly detection, and semantic classification across sites. See the research, architecture, and benchmarks pages for more.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: