# Model merging Model merging combines several fine-tuned versions of one base model into a single model by arithmetic on their weights, with averaging, TIES, DARE and SLERP. ## Overview Fine-tuning a large model for one skill is now routine. A team might have one version of a base model tuned for coding, another for a particular language, and a third for following instructions politely. The obvious question is whether you can get all three skills in one model without training again on all the data. Model merging answers that question, often surprisingly well. Model merging combines models directly in weight space. Each model is a long list of numbers, and merging computes a new list from them: an average, a weighted sum, or a more careful combination that handles conflicts between the models. The result is a single model of the same size as the inputs, with no extra cost at inference time. No gradient training is needed, so a merge can run in minutes on modest hardware, and many open-weight models published on [Hugging Face](/tools/hugging-face) are merges of other models. An everyday analogy: imagine several cooks who each started from the same basic sauce recipe and then adjusted it, one adding more garlic, another more chilli, a third more herbs. Because they all began from the same base, you can write down each cook's changes as a list of adjustments and combine them into one recipe. If two cooks pulled the salt in opposite directions, you must decide whose change wins. If the cooks had started from completely different recipes, though, averaging their ingredient lists would produce something inedible. Model merging works for the same reason, and fails for the same reason. ## What it is | Method | Core idea | Typical use | |---|---|---| | Linear averaging and model soups | Average the weights of several fine-tunes | Robustness, combining hyperparameter runs | | Task arithmetic | Add or subtract task vectors (fine-tuned minus base weights) | Adding skills, removing behaviours | | TIES-Merging | Trim small changes, elect a sign per parameter, merge agreeing values | Merging several task-specific models | | DARE | Randomly drop most delta parameters and rescale the rest | Sparsifying deltas before merging | | SLERP | Interpolate two models along a sphere rather than a straight line | Smooth blend of exactly two models | | Fisher-weighted merging | Weight each parameter by how important it is to each model | Principled weighted averaging | | Evolutionary merging | Search over merge recipes with an evolutionary algorithm | Automatically finding good combinations | | Passthrough (layer stacking) | Concatenate layers from different models | Building deeper "frankenmerge" models | Most methods work on task vectors. A task vector is the difference between a fine-tuned model's weights and the base model's weights. It captures what fine-tuning changed. Merging methods then decide how to combine several task vectors and add the result back to the base. The central requirement is a shared starting point. Merging works best, and often only works at all, when the models were fine-tuned from the same pretrained checkpoint, share the same architecture and tokenizer, and have not drifted too far from it. Models trained from different random initialisations generally cannot be averaged directly, because their internal units are arranged differently. ## Where it came from Averaging weights has a long history within a single training run. Pavel Izmailov and colleagues introduced stochastic weight averaging in 2018, showing that averaging checkpoints along a training trajectory finds flatter, better-generalising solutions. In 2020, Jonathan Frankle and colleagues studied linear mode connectivity: whether the straight line between two networks' weights stays low in loss. They found this holds when networks share enough of their early training, which explains why models fine-tuned from one pretrained checkpoint can be averaged. Merging separate fine-tuned models took off in 2022. In "Model soups" (ICML 2022), Mitchell Wortsman and colleagues averaged the weights of models fine-tuned with different hyperparameters and reported a new state of the art of 90.94 per cent top-1 accuracy on ImageNet with a ViT-G model, with no extra inference cost. They described a uniform soup, which averages everything, and a greedy soup, which adds models one at a time only if held-out accuracy improves. Michael Matena and Colin Raffel proposed Fisher-weighted averaging at NeurIPS 2022. Samuel Ainsworth, Jonathan Hayase and Siddhartha Srinivasa's "Git Re-Basin" (ICLR 2023) showed that permuting hidden units to align two independently trained networks can make them mergeable. Gabriel Ilharco and colleagues introduced task arithmetic in "Editing Models with Task Arithmetic" (ICLR 2023). They showed that adding task vectors can give a model several skills at once, negating a task vector can remove a behaviour while barely affecting others, and analogies between tasks can be exploited. Prateek Yadav and colleagues tackled interference between task vectors with TIES-Merging (NeurIPS 2023). In November 2023, Le Yu and colleagues introduced DARE in "Language Models are Super Mario" (ICML 2024), finding that 90 or even 99 per cent of fine-tuning deltas could be dropped and the rest rescaled with little loss. In March 2024, Takuya Akiba and colleagues at Sakana AI published "Evolutionary Optimization of Model Merging Recipes", which used evolutionary search to combine models both in parameter space and in the order of layers. It produced EvoLLM-JP, a Japanese language model with maths reasoning, and was published in Nature Machine Intelligence in January 2025. The same month, Charles Goddard and colleagues at Arcee described mergekit, an open-source toolkit for merging large language models, in a paper presented at EMNLP 2024. SLERP itself is much older: Ken Shoemake introduced spherical linear interpolation for animating rotations in a 1985 SIGGRAPH paper. ## How it works ### The key formulas ```text Linear average: θ_merged = Σᵢ αᵢ · θᵢ , Σᵢ αᵢ = 1 Task vector: τᵢ = θᵢ − θ_base Task arithmetic: θ_merged = θ_base + λ · Σᵢ τᵢ DARE: τ̃ᵢ = (mᵢ ⊙ τᵢ) / (1 − p), mᵢ ~ Bernoulli(1 − p) SLERP: θ(t) = sin((1 − t)Ω)/sin Ω · θ₁ + sin(tΩ)/sin Ω · θ₂ , cos Ω = θ₁·θ₂ / (‖θ₁‖‖θ₂‖) ``` In the first line, θᵢ are the weights of the i-th model and αᵢ are mixing weights that add up to one. In the second, τᵢ is the task vector of model i, its weights minus the base weights θ_base. In the third, λ is a scaling factor that controls how strongly the summed task vectors are applied. In the DARE line, p is the drop rate, mᵢ is a random mask that keeps each delta with probability 1 − p, ⊙ is element-wise multiplication, and dividing by 1 − p rescales the survivors so the expected change is preserved. In the SLERP line, θ₁ and θ₂ are two models' weight vectors (usually per layer), t between 0 and 1 sets the blend, and Ω is the angle between the two vectors. ### Linear averaging and soups The simplest merge averages corresponding weights. It works when the models lie in the same low-loss basin, which is typical for fine-tunes of one checkpoint with different seeds or hyperparameters. The greedy soup adds a safeguard: it sorts models by validation accuracy and adds each to the average only if accuracy improves. ### Task arithmetic Task arithmetic adds scaled task vectors to the base. A scaling factor below one is common, because adding several full task vectors can overshoot. Negation, subtracting a task vector, has been used to reduce unwanted behaviours, although it must be checked carefully for side effects. ### TIES: resolving interference When several task vectors are summed, two kinds of interference appear. Many parameters change only slightly, adding noise. Others change in opposite directions in different models, so they cancel. TIES handles this in three steps: trim each task vector to keep only its largest-magnitude changes; elect a sign for each parameter based on the combined magnitude of positive and negative changes; and merge by averaging only the values that agree with the elected sign. The merged vector is then added to the base. ### DARE: drop and rescale DARE observes that fine-tuning deltas, especially from [supervised fine-tuning](/tools/sft), are highly redundant. It randomly drops a large fraction of each task vector and rescales the remainder. On its own, this compresses deltas; combined with TIES-style sign handling or simple addition, it reduces interference when merging several models. ### SLERP SLERP blends two models along the arc between their weight vectors rather than the straight line. Linear interpolation between two vectors shrinks their magnitude in the middle when they point in different directions; SLERP preserves it. It is limited to two models at a time, and practitioners often vary t by layer type, for example using different blends for attention and feed-forward layers. ### Evolutionary merging Choosing weights, densities and scaling factors by hand is guesswork. Evolutionary merging treats the merge recipe as a set of parameters and searches it with an evolutionary algorithm, scoring each candidate on a validation task. Sakana AI's work also searched the data flow space, deciding which layers from which model should be used in what order, which allows merging models with different specialities into new structures. ### A mergekit configuration mergekit, licensed under LGPL-3.0, describes merges in YAML and runs them from the command line with `mergekit-yaml config.yml ./output-model`. It loads only the tensors needed at each step, so merges can run on a CPU with limited memory. A TIES merge of two fine-tunes looks like this; the model names are placeholders: ```yaml models: - model: example-org/base-model - model: example-org/base-model-code-ft parameters: density: 0.5 weight: 0.5 - model: example-org/base-model-chat-ft parameters: density: 0.5 weight: 0.5 merge_method: ties base_model: example-org/base-model parameters: normalize: true dtype: bfloat16 ``` Here `density` is the fraction of each task vector kept after trimming and `weight` scales its contribution. mergekit also supports linear, SLERP, task arithmetic, DARE variants, model breadcrumbs, model stock, passthrough and several other methods, and can extract a [LoRA](/tools/lora) adapter from the difference between a fine-tuned model and its base. ## Key concepts and terms | Term | Meaning | |---|---| | Weight space | The space of all possible values of a model's parameters | | Base model | The shared pretrained checkpoint the merged models were fine-tuned from | | Task vector | Fine-tuned weights minus base weights | | Model soup | An average of several fine-tuned models' weights | | Interference | Task vectors cancelling or adding noise when combined | | Sign conflict | Different models moving the same parameter in opposite directions | | Density | Fraction of a task vector kept after trimming or dropping | | Linear mode connectivity | Low loss along the straight line between two models | | Permutation symmetry | Reordering hidden units without changing the function | | Frankenmerge | A model built by stacking layers from different models | | Merge recipe | The full specification of models, method and parameters for a merge | ## A worked example Consider an illustrative team with an open-weight 7-billion-parameter base model. They have three fine-tunes of it: one trained on their internal code, one on support conversations in Turkish, and one on general instruction following. Each fine-tune is good at its own job and noticeably worse at the others. First they try a uniform linear average of the three. The merged model is better at Turkish support than the code model and better at code than the support model, but worse than each specialist on its own task. Their evaluation shows code accuracy dropping from 62 per cent for the specialist to 48 per cent for the average. Next they try TIES with a density of 0.5 and equal weights. Trimming removes the many tiny changes, and sign election stops the code and Turkish fine-tunes from cancelling each other in shared layers. Code accuracy in the merge recovers to 57 per cent, Turkish support quality is close to the specialist, and instruction following is preserved. Adding DARE with a drop rate of 0.9 before the TIES step makes little difference here, so they keep the simpler recipe. Finally, they run a small search over weights and densities, scoring each candidate on a combined validation set, in the spirit of evolutionary merging. The best recipe gives the code model a slightly higher weight. The final merge does not beat each specialist on its own task, but it comes within a few points on all three, in one model with no routing and no extra serving cost. They check safety behaviour separately, because merges can weaken refusals learned by one of the parents. The numbers are illustrative; real results vary widely by model and task. ## Strengths and limitations **Strengths:** - No gradient training: merges run quickly, often on a CPU, and cost little. - No extra inference cost: the merged model is the same size as each parent. - Combines skills without needing the original training data, which may be private or lost. - Model soups can improve accuracy and robustness over any single fine-tune. - Easy to explore many combinations and to automate the search. **Limitations:** - Requires compatible models: same architecture and tokenizer, and usually the same base checkpoint. - Interference can reduce each skill compared with the specialist model. - Results are empirical and hard to predict; a recipe that works for one model family may fail for another. - Merged models can lose safety behaviours or formatting conventions from one parent. - Leaderboard gains from merges can reflect overfitting to the benchmark used to choose the recipe. - Merging cannot add knowledge that none of the parents has. ## How it compares with alternatives | Approach | Training needed | Inference cost | Main trade-off | |---|---|---|---| | Model merging | None (or a cheap search) | One model | Interference, compatibility limits | | Multi-task [fine-tuning](/tools/fine-tuning) | Full training on all data | One model | Needs all data and compute | | [Ensemble learning](/tools/ensemble-learning) | None beyond the parents | Runs every model | Higher serving cost, usually robust | | Adapter switching ([PEFT](/tools/peft)) | Train one adapter per task | One base plus small adapters | Must choose the adapter per request | | [LLM routing](/tools/llm-routing) | Router training | One model per request | Router errors, several models to serve | | [Mixture of experts](/tools/mixture-of-experts) | Full training | Sparse activation | Large total size, complex training | Merging sits between ensembles and multi-task training. An ensemble averages predictions and usually performs better, but costs as much as running every model. Multi-task fine-tuning usually gives the best single model, but needs all the data and compute. Merging is the cheapest of the three and often captures a large share of the benefit. ## Practical guidance and best practices 1. Merge only models that share a base checkpoint, architecture and tokenizer; check the configuration files before starting. 2. Build an evaluation set that covers every skill you care about, plus safety and formatting checks, before you merge anything. 3. Start simple: a linear average or task arithmetic with a scaling factor below one, then try TIES or DARE if interference appears. 4. Keep the base model in the merge for task-vector methods; it is the reference point that makes deltas meaningful. 5. Sweep the key parameters, such as weights, density and scaling, and choose them on a validation set separate from your final test set. 6. Use SLERP when blending exactly two models, and consider different blends for different layer types. 7. Compare the merge with each parent and with a simple baseline, and only ship it if it wins on your priorities. 8. Record the full merge recipe, including model versions and tool version, so the merge can be reproduced. 9. Check licences: a merged model inherits the obligations of every parent. ## Common mistakes - Averaging models trained from different base checkpoints and getting a broken model. - Choosing the recipe on the same benchmark used to report results, which overstates the gains. - Forgetting to check the tokenizer and chat template, so the merged model formats conversations incorrectly. - Assuming a merge preserves safety behaviour without testing it. - Applying full-strength task vectors from many models at once, which overshoots and degrades everything. - Ignoring licence terms of one of the parent models. ## Frequently asked questions ### Why does averaging weights work at all? Fine-tuned models that start from the same pretrained checkpoint tend to stay in the same broad low-loss region of weight space. Research on linear mode connectivity shows that the straight line between such models often stays low in loss, so points along it, including the average, behave sensibly. ### Can I merge models with different architectures? Not with standard weight-space methods. They need matching shapes, which means the same architecture and usually the same tokenizer. Techniques that align permutations can help for independently trained models of the same architecture, and evolutionary data-flow merging can combine layers in new ways, but general cross-architecture merging is not a solved problem. ### What is the difference between TIES and DARE? TIES reduces interference by keeping only the largest changes in each task vector and resolving sign conflicts before averaging. DARE randomly drops most of each task vector and rescales what remains. They address related problems and are often combined, as in the DARE-TIES method. ### Will a merged model beat the specialist models? Usually not on each specialist's own task. The typical outcome is a single model that comes close to several specialists at once. In some cases, especially model soups of the same task, the merge does beat every individual model. ### Do I need a GPU to merge models? Often not. mergekit, for example, loads only the tensors needed for each operation and is designed to run even on a laptop without a GPU, although a GPU speeds things up. Evaluating the merged model is usually the more expensive part. ### Is merging the same as ensembling? No. Ensembling runs several models and combines their predictions, which multiplies inference cost. Merging combines the weights into one model, so inference costs the same as a single model. ## Glossary - Model merging: combining model weights into one model without gradient training. - Task vector: the difference between fine-tuned and base weights. - Model soup: a weight average of several fine-tuned models. - Task arithmetic: adding or subtracting scaled task vectors. - TIES-Merging: trim, elect sign and merge to reduce interference. - DARE: drop and rescale delta parameters. - SLERP: spherical interpolation between two weight vectors. - Evolutionary merging: searching merge recipes with an evolutionary algorithm. - Frankenmerge: stacking layers from different models into a deeper one. - mergekit: an open-source toolkit for merging language models. ## Summary Model merging builds a new model by arithmetic on the weights of existing ones, usually fine-tunes of the same base model. Simple averaging and model soups work because such models share a low-loss region; task arithmetic treats fine-tuning changes as vectors that can be added or removed; TIES and DARE reduce the interference that appears when several task vectors meet; SLERP blends two models smoothly; and evolutionary merging searches for recipes automatically. Tools such as mergekit make merges fast and cheap, even without a GPU. Merging works best with compatible models, careful evaluation across every skill and safety behaviour, and honest testing on data that was not used to choose the recipe.