# Weight initialization Weight initialization sets a network's starting weights so signals neither vanish nor explode. Learn zero, random, Xavier, He, orthogonal and LSUV methods in PyTorch. ## Overview Before a [neural network](/tools/neural-networks) learns anything, every weight needs a starting value. That choice, called weight initialization, has a surprisingly large effect. Start with values that are too small and the signal fades as it passes through each layer, so the output is nearly constant and the gradients that reach early layers are tiny. Start too large and the signal grows layer after layer until numbers overflow. Start with every weight equal and all neurons in a layer stay identical forever. Good initialization avoids all three, so that training with [gradient descent](/tools/gradient-descent) can begin from a healthy place. An everyday picture is a long chain of people passing a message by whispering. If everyone whispers a bit more quietly than they heard, the message is inaudible by the tenth person. If everyone speaks a bit louder, it becomes a shout. If everyone says exactly the same word whatever they hear, nothing useful passes at all. You want each person to pass the message on at roughly the volume they received it, with their own small variation. Initialization schemes are rules for setting that volume layer by layer. This page explains why zero initialization fails, how plain random initialization goes wrong in deep networks, and how Xavier (Glorot), He (Kaiming), orthogonal and LSUV initialization keep signals in range. It also covers biases, the interaction with [activation functions](/tools/activation-functions) and [normalization layers](/tools/normalization-layers), and how to apply all this with PyTorch's `nn.init`. ## What it is Initialization is a rule for drawing each starting weight, usually from a normal or uniform distribution with zero mean and a carefully chosen spread. The spread depends on the layer's shape. | Concept | Meaning | |---|---| | Fan-in | Number of inputs feeding each neuron; for a dense layer, the input size | | Fan-out | Number of outputs each input feeds; for a dense layer, the output size | | Gain | A multiplier that corrects the scale for a particular activation function | | Variance preservation | Keeping the variance of activations (forward) or gradients (backward) roughly constant across layers | | Symmetry breaking | Making neurons different from each other so they learn different features | | Uniform vs normal | Two distribution shapes; both work if the variance is matched | | Truncated normal | A normal distribution with extreme values redrawn, used in many Transformers | | Bias initialization | Usually zero; sometimes a chosen constant for a specific reason | For a convolutional layer, fan-in is the number of input channels times the kernel area, and fan-out is the number of output channels times the kernel area. ## Where it came from Early networks used small random weights chosen by rule of thumb. In 1998 Yann LeCun, Léon Bottou, Genevieve Orr and Klaus-Robert Müller's "Efficient BackProp" recommended scaling weights by the inverse square root of the fan-in, the basis of what is now called LeCun initialization. In 2010 Xavier Glorot and Yoshua Bengio published "Understanding the difficulty of training deep feedforward neural networks" at AISTATS. They analysed how activation and gradient variances change across layers and proposed balancing fan-in and fan-out, now known as Xavier or Glorot initialization. In 2013 Andrew Saxe, James McClelland and Surya Ganguli studied learning dynamics in deep linear networks and showed the benefits of random orthogonal weight matrices. In 2015 Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun's "Delving Deep into Rectifiers" derived the correct scale for ReLU networks, now called He or Kaiming initialization, and trained much deeper rectifier networks from scratch. Also in 2015, Dmytro Mishkin and Jiří Matas proposed layer-sequential unit-variance (LSUV) initialization in "All you need is a good init" (ICLR 2016), which sets scales from data rather than formulas. Batch normalization, published by Sergey Ioffe and Christian Szegedy in 2015, reduced but did not remove sensitivity to initialization. Later work such as Fixup (Hongyi Zhang, Yann Dauphin and Tengyu Ma, ICLR 2019) showed that careful initialization alone can train very deep residual networks without normalization layers. ## How it works ### Why zero (or constant) initialization fails If every weight in a layer is zero, every neuron in that layer computes the same output. During backpropagation each receives the same gradient and makes the same update. They stay identical forever, so a layer of 512 neurons behaves like one neuron copied 512 times. With ReLU and zero weights it is even worse: outputs and gradients are both zero, so nothing moves. Randomness is needed to break this symmetry. Biases, by contrast, can safely start at zero, because the random weights already make neurons differ. ### Why plain random initialization is not enough Random weights break symmetry, but their scale matters. Each neuron adds up `fan_in` terms. If each weight has variance `Var(w)` and each input has variance about 1, the sum has variance about `fan_in × Var(w)`. Over many layers this factor is multiplied again and again. If it is a little below 1, activations shrink towards zero exponentially; if a little above, they blow up. The same thing happens to gradients flowing backwards, causing vanishing or exploding gradients. The fix is to choose `Var(w)` so the factor is about 1. ### The main formulas ```text LeCun: Var(w) = 1 / fan_in Xavier/Glorot: Var(w) = 2 / (fan_in + fan_out) uniform bound = gain · sqrt( 6 / (fan_in + fan_out) ) He/Kaiming: Var(w) = 2 / fan_in (ReLU, mode = fan_in) normal std = gain / sqrt(fan_mode), gain = sqrt(2) for ReLU Gain values: linear, sigmoid = 1 ; tanh = 5/3 ; ReLU = sqrt(2) ; Leaky ReLU = sqrt( 2 / (1 + a²) ) ; SELU = 3/4 Orthogonal: W = gain · Q, where Q has orthonormal rows or columns ``` Here `Var(w)` is the variance of each weight, `fan_in` and `fan_out` are the numbers of inputs and outputs of the layer, `gain` is the activation-specific multiplier, `fan_mode` is whichever of fan-in or fan-out you choose to preserve, `a` is the negative slope of Leaky ReLU, and `Q` is a random orthogonal matrix, for example from a QR decomposition of a Gaussian matrix. ### Xavier (Glorot) Glorot and Bengio wanted to keep variance steady in both the forward pass (which depends on fan-in) and the backward pass (which depends on fan-out). You cannot satisfy both exactly unless they are equal, so they used the average. The derivation assumes an activation that behaves roughly linearly near zero, such as tanh or sigmoid around the origin. Xavier is the usual choice for tanh, sigmoid and linear layers, and it is common in attention projections. ### He (Kaiming) ReLU sets about half its inputs to zero, which halves the variance of its output. He initialization doubles the weight variance to compensate, giving `2 / fan_in`. With `mode='fan_in'` it preserves the forward signal; with `mode='fan_out'` it preserves gradients in the backward pass. Both work well in practice. It is the standard choice for ReLU, Leaky ReLU and similar activations in [CNNs](/tools/cnn) and MLPs. ### Orthogonal An orthogonal matrix rotates vectors without stretching or shrinking them. Using one (times a gain) as the starting weight keeps the length of signals exactly in linear networks and approximately in non-linear ones. It is especially popular for the recurrent weight matrix in [LSTM](/tools/lstm) and [GRU](/tools/gru) cells, where the same matrix is applied at every time step and small scale errors compound quickly. ### LSUV in general terms Layer-sequential unit-variance initialization uses data instead of a formula. Start with orthogonal weights. Then, layer by layer from input to output, pass a mini-batch through the network, measure the variance of that layer's output, and divide the layer's weights by the square root of that variance. Repeat until the variance is close to 1, then move to the next layer. Because it measures what actually happens, it adapts to unusual activations and architectures that formulas do not cover. ### Bias initialization Biases usually start at zero. Some useful exceptions: - **LSTM forget gate.** Setting its bias to 1 makes the cell remember by default at the start of training; Jozefowicz, Zaremba and Sutskever (2015) found this helped. - **Imbalanced classification outputs.** Setting the final bias to `log(p / (1 − p))`, where `p` is the positive-class rate, makes the initial prediction match the base rate. The focal loss paper used this idea with a prior of 0.01. - **Regression outputs.** Setting the final bias to the target mean makes the starting prediction sensible. - **ReLU layers.** A small positive bias is sometimes used to reduce early dead units, though zero is the common default. ### Interaction with normalization and residual connections [Normalization layers](/tools/normalization-layers) such as batch norm and layer norm rescale activations during the forward pass, so they reduce sensitivity to the exact weight scale. They do not make initialization irrelevant: the first layers before any normalization, the residual branches and the output layer still matter. In residual networks the outputs of many branches add up, so variance grows with depth. Common fixes are to initialise the last normalization scale (gamma) in each residual block to zero, as in Goyal and colleagues' 2017 large-batch ImageNet work and torchvision's `zero_init_residual` option, or to shrink residual-branch weights with depth, as GPT-2 did by scaling them by one over the square root of the number of residual layers. Many Transformers, including BERT, initialise weights from a truncated normal with standard deviation 0.02. ### Defaults and PyTorch PyTorch already initialises its layers. `nn.Linear` and `nn.Conv2d` use `kaiming_uniform_` with `a=sqrt(5)`, which works out to a uniform distribution between `−1/sqrt(fan_in)` and `1/sqrt(fan_in)`, and draw biases from the same range. This is a reasonable default but not exactly He initialization for ReLU, so many practitioners reset weights explicitly. ```python import torch from torch import nn def init_weights(m): if isinstance(m, (nn.Linear, nn.Conv2d)): nn.init.kaiming_normal_(m.weight, mode="fan_in", nonlinearity="relu") if m.bias is not None: nn.init.zeros_(m.bias) elif isinstance(m, nn.LSTM): for name, p in m.named_parameters(): if "weight_hh" in name: nn.init.orthogonal_(p) elif "weight_ih" in name: nn.init.xavier_uniform_(p) elif "bias" in name: nn.init.zeros_(p) h = p.shape[0] // 4 # gates: input, forget, cell, output p.data[h:2 * h].fill_(1.0) # forget-gate bias = 1 model = nn.Sequential(nn.Linear(128, 512), nn.ReLU(), nn.Linear(512, 10)) model.apply(init_weights) print(nn.init.calculate_gain("tanh")) # 5/3 ≈ 1.667 ``` Note that PyTorch's LSTM has two bias vectors per layer (`bias_ih` and `bias_hh`), which are added together, so this code sets the effective forget-gate bias to 2; set it on only one vector if you want exactly 1. ## Key concepts and terms | Term | Meaning | |---|---| | Initialization | Choosing starting values for parameters before training | | Symmetry breaking | Using randomness so neurons in a layer learn different things | | Fan-in / fan-out | Number of inputs to, or outputs from, each unit of a layer | | Variance preservation | Keeping signal or gradient variance steady across layers | | Gain | Activation-specific scale factor applied to an initializer | | Xavier / Glorot | Variance `2 / (fan_in + fan_out)`; suited to tanh, sigmoid, linear | | He / Kaiming | Variance `2 / fan_in`; suited to ReLU family | | LeCun | Variance `1 / fan_in`; used with SELU | | Orthogonal initialization | Starting weights from a scaled orthogonal matrix | | LSUV | Data-driven rescaling of each layer to unit output variance | | Zero-init residual | Starting each residual branch's final scale at zero | ## A worked example An illustrative ten-layer MLP has 512 units per layer and ReLU activations. For ReLU, each layer multiplies the activation variance by about `fan_in × Var(w) / 2`. Compare four choices. - **Normal with std 0.01.** Variance 0.0001, factor 512 × 0.0001 ÷ 2 = 0.0256. After ten layers the signal variance is about 0.0256¹⁰ ≈ 1.2 × 10⁻¹⁶. The output is effectively constant and gradients reaching the first layer are negligible. - **Normal with std 0.1.** Variance 0.01, factor 2.56. After ten layers the variance is about 2.56¹⁰ ≈ 12,000. Activations explode and the loss is likely to be enormous or NaN in the first steps. - **Xavier (fan-in = fan-out = 512).** Variance 2 ÷ 1,024 ≈ 0.00195, factor 0.5. After ten layers the variance is 0.5¹⁰ ≈ 0.001. Trainable at this depth but already shrinking; deeper networks would suffer. - **He (fan-in 512).** Variance 2 ÷ 512 ≈ 0.0039, standard deviation about 0.0625, factor exactly 1. The variance stays near its input value through all ten layers. The difference between 0.01 and 0.0625 looks tiny on paper. Compounded over ten layers, it is the difference between a network that cannot learn and one that trains from the first step. ## Strengths and limitations **Strengths:** - Costs nothing at training time and often decides whether a deep network trains at all. - Principled formulas exist for the most common activations. - Reduces the need for very small learning rates at the start of training. - Data-driven methods such as LSUV handle architectures the formulas do not. - Good residual initialization enables very deep networks, even without normalization. **Limitations:** - Formulas assume independent, zero-mean inputs and specific activations; real networks only approximately satisfy this. - Preserving forward and backward variance at the same time is only exact when fan-in equals fan-out. - New layer types (attention, gating, mixture-of-experts) may need their own scaling rules. - Good initialization does not fix a bad learning rate, poor data or the wrong architecture. - Framework defaults vary between libraries and versions, which can cause surprises when porting code. ## How it compares with alternatives | Approach | How it controls scale | Best fit | Drawback | |---|---|---|---| | Xavier / Glorot | Formula from fan-in and fan-out | tanh, sigmoid, linear, attention | Under-scales ReLU networks | | He / Kaiming | Formula from fan-in (or fan-out) | ReLU-family CNNs and MLPs | Assumes ReLU-like activation | | Orthogonal | Exact norm preservation at start | Recurrent weights, very deep linear parts | Extra computation at setup | | LSUV | Measured variance on real data | Unusual or custom architectures | Needs a forward pass on data before training | | [Normalization layers](/tools/normalization-layers) | Rescale activations during training | Most deep networks | Extra compute; not a full substitute | ## Practical guidance and best practices 1. Match the scheme to the activation: He for ReLU, Leaky ReLU, GELU and SiLU; Xavier for tanh, sigmoid and linear layers; LeCun normal for SELU. 2. Start biases at zero unless you have a specific reason, such as an LSTM forget gate or an imbalanced output. 3. Initialise the final classification bias to the log-odds of the base rate when classes are very imbalanced. 4. For residual networks, zero-initialise the last normalization scale in each block or scale down residual branches with depth. 5. Use orthogonal initialization for recurrent weight matrices. 6. After building a model, run one batch and print the mean and standard deviation of each layer's activations; they should stay roughly constant. 7. When fine-tuning a pre-trained model, only newly added layers need initializing; leave pre-trained weights alone (see [transfer learning](/tools/transfer-learning)). 8. Pair good initialization with a short learning-rate warm-up (see [learning rate schedules](/tools/learning-rate-schedules)) for large models. ## Common mistakes - Initialising all weights to zero or to the same constant. - Using Xavier with deep ReLU networks and attributing the slow start to the optimiser. - Using the old `nn.init` names without the trailing underscore; the underscore versions (such as `kaiming_normal_`) are the current in-place functions. - Overwriting pre-trained weights by applying an init function to the whole model during fine-tuning. - Ignoring that PyTorch's LSTM has two bias vectors when setting forget-gate bias. - Assuming batch normalization makes initialization irrelevant. ## Frequently asked questions ### Why can biases start at zero when weights cannot? Because the symmetry problem comes from neurons being identical. Random weights already make each neuron compute something different, so equal (zero) biases do not cause identical neurons. Zero biases also add no extra variance, keeping the scale analysis simple. ### Should I use the uniform or normal version? Either, if the variance matches. Uniform versions draw from a fixed range and have no extreme values; normal versions occasionally draw larger weights. Differences in final results are usually small. Many Transformers use a truncated normal to avoid rare extreme weights. ### Do I still need careful initialization with batch norm or layer norm? It matters less, but still matters. Normalization fixes scale inside the network, but not before the first normalization layer, inside residual branches, or at the output. Very deep models still benefit from residual-aware initialization. ### What is the difference between fan-in and fan-out modes in He initialization? Fan-in mode keeps the variance of activations steady in the forward pass. Fan-out mode keeps the variance of gradients steady in the backward pass. He and colleagues showed either is enough to train well in their setting. Fan-in is the PyTorch default. ### Does initialization matter for fine-tuning? For pre-trained weights, no: you keep them. For newly added heads or adapters it does. Methods such as [LoRA](/tools/lora) deliberately initialise one of their two matrices to zero so the adapted model starts exactly equal to the original. ### What does PyTorch do if I do nothing? It applies a built-in default per layer type. For `nn.Linear` and convolutions this is a Kaiming-uniform variant that gives a uniform range of plus or minus one over the square root of the fan-in. It is often acceptable, but explicit He initialization is a common improvement for deep ReLU networks. ## Glossary - Weight initialization: the rule for setting starting weight values. - Fan-in: the number of inputs to each unit in a layer. - Fan-out: the number of outputs each input connects to. - Gain: a scale factor that adapts an initializer to an activation function. - Xavier initialization: variance `2 / (fan_in + fan_out)`. - He initialization: variance `2 / fan_in`, designed for ReLU. - Orthogonal matrix: a matrix whose rows and columns are perpendicular unit vectors. - LSUV: layer-sequential unit-variance, a data-driven scaling method. - Symmetry breaking: making neurons start different so they learn different features. - Exploding gradient: gradients growing very large as they pass through layers. ## Summary Weight initialization chooses where training starts, and in deep networks that choice decides whether signals and gradients survive the trip through many layers. Zero or constant weights fail because identical neurons stay identical. Plain random weights break symmetry but shrink or blow up signals unless their variance is matched to the layer size. Xavier initialization balances fan-in and fan-out for tanh-like activations, He initialization doubles the variance for ReLU, orthogonal initialization preserves norms exactly and suits recurrent weights, and LSUV measures real activations to set each layer's scale. Biases usually start at zero, with useful exceptions for forget gates and imbalanced outputs. Normalization layers and residual-aware tricks reduce sensitivity but do not remove it, so a quick check of activation statistics after building a model is always worthwhile.