Spiking neural networks

Overview

Spiking neural networks, often shortened to SNNs, are neural networks whose neurons communicate with short, discrete events called spikes. A neuron collects input over time, and when its internal voltage crosses a threshold, it fires a spike and resets. Information is carried by when spikes occur and how many occur, not by a continuous number passed between layers.

This design brings the model closer to biological brains and enables event-driven computation: nothing happens unless a spike arrives. On suitable neuromorphic hardware, that can cut energy use sharply. The price is that spikes are not differentiable, so training needs special methods, and accuracy on many standard tasks still trails conventional networks. SNNs shine on event-based sensors, temporal signals and low-power devices. Useful companions are snnTorch, SpikingJelly, Norse and Intel Loihi.

What it is

A different kind of neuron

In a standard artificial network, a neuron computes a weighted sum of inputs, applies a function such as ReLU and passes a real number on. This happens at every layer for every input, all at once, with no sense of time.

In a spiking network, a neuron has a membrane voltage that changes over time. Input spikes push the voltage up or down. The voltage slowly leaks back to rest. If it reaches a threshold, the neuron emits a spike and the voltage resets. The output is a train of spikes over time.

An everyday picture

Think of a bucket with a small hole in the bottom. Each input spike is a cup of water poured in. Water leaks out slowly. If cups arrive fast enough, the bucket fills and tips over. That tipping is a spike, after which the bucket is emptied. A single cup does nothing. A rapid burst of cups makes the bucket tip.

This captures why timing matters: the same number of inputs can cause a spike or not, depending on how closely they arrive.

Why spikes at all

• Sparsity: most neurons are silent most of the time, so most multiplications are skipped.
• Time as a resource: the pattern of spike times can carry information in addition to rates.
• Hardware fit: spikes map onto asynchronous circuits that use power only when events occur.
• Biological realism: useful for studying how real neural circuits might compute.

The main ingredients

• Spiking neuron models such as leaky integrate-and-fire.
• Synapses that turn spikes into voltage changes, often with delays or their own dynamics.
• A way to encode inputs as spikes.
• A way to decode output spikes into predictions.
• A learning rule, which can be gradient based or local.

What it is not

• It is not a drop-in replacement for a standard deep network. The tools and tricks differ.
• It is not automatically more accurate or more efficient. Gains depend on sparse activity and suitable hardware.
• It is not limited to vision. Audio, sensor data and control tasks fit well.

Where it came from

Biological roots

In the early 1950s, a mathematical model of the squid giant axon, known as the Hodgkin-Huxley model, described how voltage spikes arise from ion channels. Simpler models followed. The integrate-and-fire idea, much older still, reduces a neuron to a leaky accumulator with a threshold. These models are the basis of today's spiking units.

Early artificial networks

The first generation of artificial networks used threshold units. The second generation used smooth activations and gradient descent and led to modern deep learning. In the 1990s, researchers proposed a third generation, spiking networks, and argued that they could be at least as powerful as sigmoid networks for some computations.

Learning rules inspired by biology

Spike-timing-dependent plasticity, or STDP, emerged as a local rule: if a presynaptic spike comes just before a postsynaptic spike, the connection strengthens, and if it comes just after, it weakens. STDP gave unsupervised learning from spike patterns and remains a core idea in neuromorphic research.

Hardware

Neuromorphic chips were designed to run spiking networks with very low power. Well-known research platforms include IBM TrueNorth, Intel Loihi and BrainScaleS. Software stacks such as Lava target this class of hardware.

Deep learning meets spikes

From the mid-2010s onward, two lines of work made SNNs trainable at scale:

• Conversion: train a normal network, then convert it into a spiking one.
• Surrogate gradients: train spiking networks directly by replacing the non-differentiable spike with a smooth stand-in in the backward pass.

Event-based cameras, which output a stream of brightness changes instead of frames, supplied natural data. Libraries such as snnTorch, SpikingJelly and Norse built on PyTorch lowered the barrier to entry.

How it works

Step 1: the neuron model

The most common model is the leaky integrate-and-fire neuron, or LIF. In discrete time, it works like this:

• The voltage decays by a factor beta each step: v = beta * v.
• The weighted input current is added: v = v + I.
• If v is at or above the threshold, emit a spike s = 1, else s = 0.
• After a spike, reset the voltage, either to zero or by subtracting the threshold.

The decay factor beta sets the memory of the neuron. A value near 1 means a long memory. A value near 0 means a neuron that forgets almost at once.

Step 2: more detailed neurons

Other models add features:

• Integrate-and-fire without leak: a perfect accumulator.
• Adaptive LIF: the threshold rises after each spike, so the neuron tires.
• Izhikevich model: a cheap model that reproduces many firing patterns.
• Hodgkin-Huxley type: detailed ion channel dynamics, too costly for most machine learning.
• Resonate-and-fire: neurons that respond to specific input frequencies.

Step 3: synapses and layers

Synapses scale spikes by weights. Some models include a synaptic current with its own decay, so a spike produces a short pulse of input rather than an instant jump. Layers can be dense, convolutional or recurrent. A convolutional spiking layer has the same weight sharing as a normal one, but its outputs are spikes.

Step 4: encode the input

Real-valued data must be turned into spikes unless the sensor already produces them. Common schemes:

Step 5: run over time

The network is simulated for T time steps. At each step, spikes travel through the layers and update voltages. The state of every neuron carries over to the next step. The cost depends on T as well as on network size.

Step 6: decode the output

Typical readouts:

• Spike count: the output neuron that fires most is the predicted class.
• First spike: the output neuron that fires first wins.
• Membrane voltage: take the final or averaged voltage of output neurons.

Using the voltage often gives smoother learning than counting spikes.

Step 7: train

There are three families of training methods.

Surrogate gradient learning

The spike function is a step: output 1 if the voltage is over the threshold, else 0. Its derivative is zero almost everywhere, and infinite at the threshold, which blocks backpropagation. The fix is to keep the hard step in the forward pass and use a smooth curve, such as a fast sigmoid or an arctangent, as the derivative in the backward pass. Backpropagation through time then trains the whole network.

An analogy: a door that is either open or shut. You cannot tell how much to push it from the door alone. So during training you pretend the door is slightly springy, and use that spring to decide how hard to push.

ANN-to-SNN conversion

Train a standard network with ReLU, then replace each ReLU with a spiking neuron whose firing rate approximates the activation. Weights and thresholds are rescaled. Conversion reaches good accuracy but often needs many time steps.

Local and biologically inspired rules

STDP and its relatives update weights using only local spike timing. They need no global error signal and suit on-chip learning, but accuracy on hard tasks is lower.

Step 8: deploy

On a GPU, an SNN is simulated step by step and often runs slower than a normal network of the same size. The benefit appears on neuromorphic hardware or on devices that skip work for silent neurons.

Key concepts and terms

Rate versus temporal codes

A rate code is like measuring how loud a crowd is by counting claps per minute. A temporal code is like reading a message from the exact gaps in a Morse code. Rate codes are robust and need many spikes. Temporal codes are fast and sparse but sensitive to noise and jitter.

A worked example

A numeric walk-through of a LIF neuron

Use beta = 0.8, threshold = 1.0, and reset by setting the voltage to zero after a spike. Input current over six steps is 0.5, 0.5, 0.5, 0.0, 0.9, 0.9.

Three consecutive moderate inputs cause a spike at step 3. A gap lets the voltage leak away. Two strong inputs at steps 5 and 6 cause a spike at step 6. The neuron responds to sustained or strong input and filters sparse weak input.

Training a small SNN with snnTorch

This example classifies vectors using rate-coded output and a surrogate gradient. It uses snnTorch on top of PyTorch.

import torch
import torch.nn as nn
import snntorch as snn
from snntorch import surrogate

beta = 0.9
spike_grad = surrogate.fast_sigmoid(slope=25)

class Net(nn.Module):
    def __init__(self, n_in=20, n_hidden=64, n_out=3):
        super().__init__()
        self.fc1 = nn.Linear(n_in, n_hidden)
        self.lif1 = snn.Leaky(beta=beta, spike_grad=spike_grad)
        self.fc2 = nn.Linear(n_hidden, n_out)
        self.lif2 = snn.Leaky(beta=beta, spike_grad=spike_grad)

    def forward(self, x, T=25):
        mem1 = self.lif1.init_leaky()
        mem2 = self.lif2.init_leaky()
        out_spikes = []
        for _ in range(T):                       # same input shown for T steps (direct encoding)
            cur1 = self.fc1(x)
            spk1, mem1 = self.lif1(cur1, mem1)
            cur2 = self.fc2(spk1)
            spk2, mem2 = self.lif2(cur2, mem2)
            out_spikes.append(spk2)
        return torch.stack(out_spikes).sum(0)    # spike counts per class

net = Net()
opt = torch.optim.Adam(net.parameters(), lr=2e-3)
loss_fn = nn.CrossEntropyLoss()

for step in range(300):
    x = torch.randn(32, 20)
    y = torch.randint(0, 3, (32,))
    opt.zero_grad()
    loss = loss_fn(net(x), y)
    loss.backward()
    opt.step()

What the code shows

• The same input is applied for T steps, so the network unrolls over time.
• Each layer holds a membrane state that persists between steps.
• The hard spike is used in the forward pass. The fast sigmoid derivative is used in the backward pass.
• The output is the spike count per class, treated as logits.

Encoding a numeric feature with latency

import numpy as np

def latency_encode(x, T=20):
    # x in [0, 1]; larger value -> earlier spike
    t = np.round((1.0 - x) * (T - 1)).astype(int)
    spikes = np.zeros((T, len(x)))
    spikes[t, np.arange(len(x))] = 1
    return spikes

print(latency_encode(np.array([0.95, 0.5, 0.05])).argmax(axis=0))   # [1 10 18]

Strong features fire early, and weak features fire late. A downstream neuron that responds to the first few spikes will mostly see the strong features. This gives fast decisions with few spikes.

Spiking layers in more detail

Dense spiking layers

A dense layer multiplies the spike vector from the previous layer by a weight matrix. Because the input is binary, this multiplication reduces to adding up the weights of the active inputs. No true multiplication is needed. This is the source of much of the claimed savings.

Convolutional spiking layers

A convolutional layer slides a small kernel over a map of spikes. Neurons at each location keep their own voltage. Pooling can be done by max over spikes or by summing.

Recurrent spiking layers

A recurrent layer feeds spikes from the layer back into itself with a delay. This creates a memory beyond the leak of a single neuron. Such layers are used for temporal tasks and for models of cortical circuits.

Normalisation and residual links

Deep spiking networks borrow tricks from standard ones. Residual links that add spikes or voltages help gradients flow. Time-aware batch normalisation keeps voltages in a useful range. Some designs add spiking self-attention, with binary queries and keys, to reduce cost.

Time steps and accuracy

More steps give a finer estimate of rates but cost more time and memory. Direct training with surrogate gradients can get good results with few steps, which is a main advantage over conversion.

A tiny design checklist

• Define the input type and choose an encoder.
• Choose the neuron model, usually LIF.
• Set the number of time steps.
• Pick the readout, such as final voltage or spike count.
• Choose a surrogate gradient.
• Monitor firing rates in each layer.
• Plan the target: GPU only, or neuromorphic hardware.

Strengths and limitations

Strengths

• Event-driven computation and sparse activity can save energy on suitable hardware.
• Natural handling of temporal signals, since state and time are built in.
• Good match for event cameras and other asynchronous sensors.
• Low latency in time-to-first-spike schemes.
• Biologically plausible, which makes them useful as models of neural circuits.
• Robust to some kinds of noise, because of thresholds and leak.
• Small, fixed memory per neuron.

Limitations

• Training is harder. The spike is not differentiable, and backpropagation through many time steps is memory hungry.
• On standard GPUs, simulation over T steps is often slower than a normal network.
• Accuracy often lags conventional networks on large static benchmarks.
• Neuromorphic hardware is specialised, with limited availability and toolchains.
• Efficiency claims depend on counting operations fairly. Spike counts, memory movement and time steps all matter.
• Fewer pre-trained models and less community knowledge than for mainstream deep learning.
• Encoding real-valued tabular features adds design choices.

Reading efficiency claims with care

A fair energy comparison counts the operations in both models, the cost of memory access, the number of time steps and whether the hardware really skips silent neurons. Claims that ignore these can mislead. A sparse SNN on a general GPU may use more energy than a compact normal network.

How it compares with alternatives

Related models

Spiking versus state-space models

A LIF neuron is a one-dimensional, leaky linear system followed by a threshold. A state-space layer is a bank of such linear systems without the threshold. The threshold is what makes spikes and sparsity, and also what makes training harder. Some recent designs mix the two, using linear recurrent layers with spiking outputs.

Role in tabular foundation models

Where SNNs and tables meet

A tabular foundation model learns from raw tables: rows and columns, mixed numeric and categorical cells, missing values and often no column names. Current leaders such as TabPFN, TabICL and FT-Transformer use attention on dense vectors and run on GPUs. Spiking networks are not part of that mainstream. Their relevance is in specific, concrete corners, and it is worth stating those honestly.

Row and column structure

A table row has no time axis. To feed it to an SNN, you must create one. Two ways are common:

• Present the same encoded row for T time steps (direct or rate coding), so time is only a computation device.
• Spread feature values across time using latency coding, so that larger values spike earlier.

Time-indexed tables, such as sensor logs, already have a time axis and are a natural fit. Each record becomes an event, and the network processes the stream with a persistent state.

Mixed types

Each type needs an encoder:

• Numeric features: population coding with several neurons per feature, each tuned to a value range, or latency coding.
• Categories: one neuron per category, or a learned embedding followed by a spike encoder.
• Booleans: a single neuron that spikes or stays silent.

The encoding choice has a large effect on accuracy, and there is no universal best option.

Missing values

SNNs offer a clean representation of missing cells: a missing feature simply produces no spikes. Silence is the default state of the network, so no placeholder value is needed. A separate indicator neuron can tell the model that the feature was missing, not merely small. This avoids having to impute before the model runs, although it does not replace imputation when values are needed as output.

No column names

If names are absent, the network must learn what each input channel means from its values. In an SNN, each channel is a separate input line. The network learns weights per line, so names are unneeded. A foundation model would, in addition, need a way to share knowledge across tables with different columns, which is a problem for any architecture, not just SNNs.

Pre-training on synthetic and real tables

Pre-training on many generated tables is what gives tabular foundation models their general skill. Doing this with an SNN is possible in principle, but the cost of unrolling over time steps makes large-scale pre-training expensive on GPUs. One practical route is to pre-train a conventional model and then distil or convert it into a spiking one for deployment at the edge. See Synthetic data generation.

In-context learning

In-context learning needs the model to read labelled example rows and adapt without weight updates. Attention does this directly. Spiking models have no standard mechanism for it. Research on spiking attention and on recurrent spiking memories exists, but it remains early. For now, SNNs are better thought of as task-specific learners that are trained once and deployed.

Continual learning

Local rules such as STDP and online updates fit streaming data, where a model learns as events arrive. This matches the continual-learning setting, with its risk of forgetting. Sparse activity can also reduce interference between tasks, since only a subset of neurons responds to any input. See Continued pre-training for the large-model side of the problem.

Imputation and anomaly detection

Spiking autoencoders can compress and reconstruct event streams, and a large reconstruction error flags an anomaly. Event-driven anomaly detectors are attractive for always-on sensors, because they only wake when something changes. For imputation in tables, standard methods are more accurate today. See Autoencoders, Anomaly detection, Imputation networks and Masked cell modeling.

Sector and semantic classification

Deciding what domain a table or column belongs to is a classification task over summaries of values. Attention-based models do this well. An SNN could serve as an efficient classifier at the edge, but there is little reason to use it for large-scale semantic tasks. See Semantic sector heads. For a view of how tabular models are built and measured, see /model/architecture, /model/research and /model/benchmarks.

An honest summary of the role

Spiking networks are a niche for tabular work. They make sense when tables arrive as streams from sensors, when the device is power limited, or when you want an event-driven anomaly detector. For static tables on a server, attention-based foundation models and tree ensembles such as XGBoost remain the better tools.

Practical guidance and best practices

Choosing between SNN and a standard model

• Pick an SNN if your data are events, your device is power constrained, or you will run on neuromorphic hardware.
• Pick a standard model if you need top accuracy on static data and run on a GPU.
• Consider training a standard model first, then converting.

Setting up training

• Start with direct input encoding and a small number of time steps, such as 8 to 25.
• Use a fast sigmoid or arctangent surrogate with a moderate slope.
• Use voltage or spike-count readout with a cross-entropy loss.
• Use Adam with a modest learning rate.
• Add batch normalisation or a similar layer in a time-aware form if depth grows.

Tuning neuron parameters

• Set beta by the time scale of your signal. Slow signals need beta close to 1.
• Make beta and threshold learnable, per layer or per neuron.
• Watch firing rates. If all neurons are silent, lower the threshold or raise the weights. If all fire at every step, raise the threshold.

Memory and speed

• Backpropagation through time stores states for every step. Use fewer steps, truncated backpropagation or checkpointing.
• Use mixed precision where the library allows.
• Use fused kernels offered by the libraries when available.

Encoding tabular data

• Normalise numeric features to a fixed range first.
• Use population coding for continuous values.
• Add a missing indicator input for each feature that can be absent.
• Compare direct and spike encodings on a validation set.

Measuring efficiency honestly

• Count spikes and synaptic operations on a held-out set.
• Include the number of time steps.
• Compare with a compact standard model, not only a large one.
• If you target hardware, measure on the hardware.

Common mistakes

• Expecting a large energy saving on a general-purpose GPU.
• Using too many time steps, which makes training slow and memory heavy.
• Using a surrogate slope that is too sharp or too flat.
• Letting the network go silent and not noticing because loss changes slowly.
• Encoding all features with the same scheme without testing alternatives.
• Ignoring input scaling, so that some features never cause spikes.
• Counting only spikes and not memory traffic when claiming efficiency.
• Comparing against a weak baseline.
• Forgetting to reset neuron states between samples.
• Using an SNN for a small static table where a tree ensemble would be faster and more accurate.

Frequently asked questions

What is a spiking neural network?

A spiking neural network is a network of neurons that pass short, all-or-nothing events called spikes instead of continuous numbers. Each neuron has a voltage that rises with input, leaks over time and triggers a spike at a threshold. Information is carried by spike counts and spike timing.

How do spiking neural networks differ from regular neural networks?

Regular networks pass real numbers between layers in one forward sweep, while spiking networks run over time steps and pass binary events. Spiking networks have built-in memory in their voltages and sparse activity. They are harder to train because the spike function has no useful gradient.

How are spiking neural networks trained?

The most common method is surrogate gradient learning, which keeps hard spikes in the forward pass and uses a smooth curve in the backward pass so that backpropagation through time works. Other methods convert a trained standard network into a spiking one, or use local rules such as spike-timing-dependent plasticity.

Are spiking neural networks more energy efficient?

They can be, when the hardware skips work for silent neurons and the activity is sparse. On a regular GPU the advantage often disappears, because the network is simulated for many time steps. Fair comparisons must count spikes, memory access and time steps.

What is a leaky integrate-and-fire neuron?

It is the standard simple spiking neuron. It adds incoming current to a voltage, lets the voltage decay each step, and fires a spike when the voltage passes a threshold, after which the voltage resets. The leak factor sets how long the neuron remembers its input.

What is a surrogate gradient?

A surrogate gradient is a smooth substitute for the derivative of the spike function, used only when computing gradients. The forward pass still produces hard spikes, but the backward pass pretends the spike was a smooth curve so that learning signals can flow. It makes deep spiking networks trainable with standard tools.

What kind of data suits spiking networks?

They suit event-based and temporal data, such as output from event cameras, audio, radar and sensor streams. They also suit low-power devices that must react to changes. Static images and tables can be used after encoding, but they gain less from the spiking design.

Can spiking neural networks work on tabular data?

Yes, after encoding each feature as spikes, for example with population or latency coding. They suit time-indexed tables such as sensor logs and can treat missing values as silence. For ordinary static tables, tree ensembles and tabular foundation models are usually more accurate and simpler.

What is neuromorphic computing?

Neuromorphic computing is the design of hardware that imitates the brain's event-driven, parallel and sparse style of computing. Chips of this kind run spiking networks with circuits that use power mainly when spikes occur. It is an active research area with a few specialised platforms.

Which libraries can I use to build spiking networks?

Several open-source libraries built on PyTorch offer neurons, surrogate gradients, encoders and datasets, including snnTorch, SpikingJelly and Norse. Hardware-oriented frameworks such as Lava target neuromorphic chips. A beginner can start with any of the PyTorch-based ones.

Glossary

• Adaptive threshold: a threshold that rises after a spike, causing neurons to tire.
• BPTT: backpropagation through time, used to train recurrent and spiking networks.
• Event camera: a sensor that reports brightness changes as asynchronous events.
• Izhikevich neuron: a cheap model that reproduces many firing patterns.
• Latency coding: encoding a value as the time of a spike.
• Leak: the gradual decay of membrane voltage.
• LIF: leaky integrate-and-fire neuron.
• Membrane potential: the voltage state of a neuron.
• Neuromorphic: describing hardware or systems modelled on the brain.
• Population coding: representing a value with a group of neurons tuned to different ranges.
• Rate coding: encoding a value as the number of spikes per time window.
• Refractory period: a short interval after a spike with reduced ability to fire.
• Spike: a brief binary event emitted by a neuron.
• STDP: a local learning rule that depends on the relative timing of spikes.
• Surrogate gradient: a smooth derivative used in place of the spike derivative.
• Synapse: the connection that carries spikes between neurons.

Summary

Spiking neural networks compute with timed, discrete spikes and leaky voltages. They bring sparsity, built-in memory and a close fit to event-based sensors and neuromorphic hardware. Surrogate gradients and conversion methods have made them trainable, and libraries such as snnTorch and SpikingJelly make them accessible.

For tabular foundation models, SNNs are a niche option: useful for streaming, time-indexed and low-power settings, with a natural treatment of missing values as silence, but not a replacement for attention-based models on static tables. Related reading: Autoencoders, Neural ODE, Mamba, /model/architecture and /model/benchmarks.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: