NCCL: fast multi-GPU communication for distributed training

Overview

NCCL, the NVIDIA Collective Communications Library, is the software that lets many NVIDIA GPUs exchange data quickly while they work on one job. Its name is usually pronounced "nickel". It provides collective operations, such as all-reduce, all-gather, reduce-scatter and broadcast, that move and combine tensors across GPUs in one server and across many servers. Frameworks such as PyTorch, DeepSpeed, Megatron-LM and inference engines such as vLLM use NCCL whenever a model is trained or served on more than one GPU.

An everyday picture: a group of accountants each adding up one pile of receipts. At the end of the day they need one grand total, and every accountant needs to know it. Shouting numbers across the room one by one would be slow. A better plan is to pass partial sums around a circle, each person adding their share, so everyone ends up with the total in a few quick rounds. NCCL organises exactly this kind of exchange among GPUs, choosing the fastest routes through the hardware.

Communication is often the hidden limit on large-scale AI. Each GPU may compute quickly, but if it must wait for data from others, the whole cluster slows down. NCCL's job is to make that waiting as short as possible.

What it is

Part	Purpose
Collective operations	All-reduce, all-gather, reduce-scatter, broadcast, reduce and all-to-all
Point-to-point operations	Send and receive between specific GPUs, used for pipeline parallelism
Communicator	A group of GPUs, called ranks, that communicate together
Topology detection	Discovers how GPUs, CPUs, NICs and switches are connected
Algorithms	Ring, tree and other patterns for moving data
Protocols	Low-level methods that trade latency against bandwidth
Transports	NVLink and PCIe within a node; InfiniBand, RoCE and TCP sockets between nodes
Network plug-ins	Interfaces for vendor networking such as cloud-specific fabrics
Environment variables	Settings such as `NCCL_DEBUG` for logging and tuning

NCCL is open source under a BSD-style licence and ships with NVIDIA's software stacks and framework builds.

Where it came from

NVIDIA released the first version of NCCL in the mid-2010s as an open-source library for fast collectives among GPUs inside one machine, at a time when deep learning researchers were starting to train on several GPUs at once. NCCL 2, released in 2017, added communication across machines over high-speed networks such as InfiniBand, which made multi-node training practical with standard frameworks.

Since then NCCL has grown with NVIDIA's hardware. It supports NVLink and NVSwitch, which connect GPUs inside servers at very high bandwidth, and in-network reduction features on suitable switches. Each new GPU generation, such as the H100 and B200, has brought NCCL updates to use faster links and larger NVLink domains.

How it works

Ranks and communicators

Each process in a distributed job usually controls one GPU and has a rank number. Before communicating, processes share a unique identifier, often through the framework's launcher, and create a communicator. Several communicators can exist, for example one per tensor-parallel group and one per data-parallel group.

Discovering topology

On start-up NCCL examines the machine: which GPUs connect by NVLink, which share a PCIe switch, which network cards sit closest to which GPUs. It builds a graph of the hardware and plans routes that use the fastest links and avoid bottlenecks.

All-reduce in a ring

The best-known algorithm is the ring all-reduce. Imagine eight GPUs, each with a gradient tensor. Each tensor is split into eight chunks. In the first phase, reduce-scatter, each GPU sends one chunk to its neighbour while receiving another, adding what it receives. After seven steps, each GPU holds the full sum of one chunk. In the second phase, all-gather, the summed chunks travel round the ring until every GPU has the complete result. Each GPU sends and receives about twice the tensor size in total, no matter how many GPUs take part, which is why rings use bandwidth efficiently.

Trees and other patterns

Rings take more steps as the number of GPUs grows, which adds latency for small messages. Tree algorithms reduce the number of steps and help at large scale. NCCL chooses algorithms and protocols automatically based on message size and topology.

Using NCCL from a framework

Most users never call NCCL directly.

import torch
import torch.distributed as dist

dist.init_process_group(backend="nccl")
rank = dist.get_rank()
torch.cuda.set_device(rank % torch.cuda.device_count())
t = torch.ones(4, device="cuda") * rank
dist.all_reduce(t)  # every rank now holds the sum over ranks

Launched with a tool such as torchrun, every process runs this script and NCCL performs the all-reduce across them.

Overlap with compute

Frameworks issue NCCL calls on separate CUDA streams so communication overlaps with computation. For example, in data parallelism, gradients for later layers are reduced while earlier layers are still computing their backward pass.

Key concepts and terms

Term	Meaning
Collective	An operation in which all ranks in a group take part
All-reduce	Combine values from all ranks and give everyone the result
All-gather	Give every rank a copy of every rank's data
Reduce-scatter	Combine values and give each rank one part of the result
All-to-all	Each rank sends distinct data to every other rank
Rank	The identifier of one participant, usually one GPU
Communicator	The group of ranks that communicate together
NVLink	NVIDIA's high-bandwidth link between GPUs
InfiniBand	A high-speed, low-latency network used in clusters
RoCE	RDMA over Converged Ethernet, an alternative cluster network
Ring algorithm	Passing chunks around a logical ring of ranks
Bus bandwidth	A normalised measure of collective performance

A worked example

This scenario is illustrative. A team trains a 7-billion-parameter model with data parallelism on 4 servers, each with 8 GPUs connected by NVLink, joined by a 400 Gb/s InfiniBand network per GPU.

Each training step produces BF16 gradients of about 14 GB per GPU, which must be all-reduced across 32 GPUs.
Inside each server, NVLink carries traffic at hundreds of gigabytes per second. Between servers, the network is the slower link, so NCCL arranges traffic to keep most data movement inside servers.
With default settings and overlap, each step spends about 15 per cent of its time waiting for communication.
A misconfiguration on one server leaves its network cards unused, and NCCL falls back to TCP sockets. Step time doubles. Setting NCCL_DEBUG=INFO shows the transport chosen, revealing the problem.
After fixing the network configuration, the team runs the NCCL performance tests, which report bus bandwidth close to the expected value for their network, confirming the cluster is healthy before a multi-week run.

Strengths and limitations

Strengths:

Highly optimised collectives for NVIDIA GPUs, within and across servers.
Automatic topology detection and algorithm selection.
Supports NVLink, NVSwitch, InfiniBand, RoCE and plug-ins for cloud networks.
Integrated into every major deep learning framework.
Open source, with performance-test tools available.

Limitations:

Designed for NVIDIA GPUs; other vendors use their own libraries.
Failures and hangs can be hard to diagnose across many nodes.
Performance depends heavily on correct network and driver configuration.
Collective calls must match across ranks; a mismatch causes deadlocks.
Fault tolerance for failing nodes is limited and handled mostly by frameworks.

How it compares with alternatives

Library	Hardware	Notes
NCCL	NVIDIA GPUs	Standard for NVIDIA training and inference
RCCL	AMD GPUs	AMD's NCCL-compatible library
Gloo	CPUs, some GPU use	Used by PyTorch for CPU collectives
MPI	General HPC	Long-standing standard, often used for launching jobs
Collectives on Cloud TPU	Google TPUs	Handled by XLA and the TPU interconnect

Practical guidance and best practices

Use the framework's NCCL backend rather than calling NCCL directly unless you need to.
Run NCCL performance tests on a new cluster before starting long jobs.
Set NCCL_DEBUG=INFO when debugging to see topology and transport choices.
Make sure each GPU uses the nearest network card and that RDMA is working.
Keep NCCL, drivers, CUDA and network software at tested, compatible versions.
Overlap communication with compute through framework options such as gradient bucketing.
Set timeouts so a stuck collective fails clearly instead of hanging forever.
Prefer layouts that keep heavy communication, such as tensor parallelism, inside NVLink domains.

Common mistakes

Running tensor parallelism across servers over a slow network.
Letting NCCL silently fall back to TCP because RDMA is misconfigured.
Calling collectives in a different order on different ranks, causing deadlocks.
Ignoring firewall or network interface settings, so ranks cannot connect.
Mixing GPU types or driver versions within one job.
Assuming slow training is a compute problem without measuring communication.

Frequently asked questions

What does NCCL do in simple terms?

It moves and combines data between GPUs so they can work together on one model, for example summing gradients from every GPU after each training step.

Do I need to install NCCL separately?

Usually not. PyTorch and other frameworks ship builds that include NCCL, and NVIDIA containers include it. Cluster operators may install specific versions for their network.

What is the difference between NVLink and InfiniBand?

NVLink connects GPUs inside a server, or within an NVLink domain, at very high bandwidth. InfiniBand connects servers to each other over a cluster network. NCCL uses both, choosing the right one for each path.

Is NCCL used for inference too?

Yes. When a large model is split across GPUs with tensor or pipeline parallelism, inference engines use NCCL to pass activations between GPUs at every layer or stage.

Why does my distributed job hang?

Common causes are ranks calling different collectives, network connectivity problems, or one process crashing. Debug logs, timeouts and checking that every rank reaches the same code are the usual first steps.

Glossary

RDMA: remote direct memory access, moving data between machines without involving the CPU.
NVSwitch: a switch chip that connects many GPUs by NVLink.
Gradient bucketing: grouping gradients into larger messages for efficient all-reduce.
Deadlock: a state where processes wait for each other forever.
torchrun: PyTorch's launcher for distributed jobs.
NIC: network interface card.
Data parallelism: training copies of a model on different data and averaging gradients.

Summary

NCCL is NVIDIA's library for collective communication among GPUs, providing all-reduce, all-gather, reduce-scatter and other operations over NVLink, PCIe, InfiniBand and Ethernet, with automatic topology detection and algorithm choice. It has been the backbone of multi-GPU and multi-node training on NVIDIA hardware since multi-node support arrived in 2017, and it also underpins distributed inference. Most users meet it through their framework, but checking its configuration and bandwidth is essential before any large job.

It belongs to Hardware in my stack, under MLOps and AI infrastructure. Others in the same group:

CUDA Cloud TPU NVIDIA H100 NVIDIA H200 NVIDIA B200