CUDA

Overview

CUDA is NVIDIA's platform and programming model for general-purpose computing on its graphics processors. It lets programmers write functions, called kernels, that run across thousands of GPU threads at once, and provides a compiler, a runtime, drivers, debugging and profiling tools, and a large set of optimised libraries. Almost every modern deep learning framework, including PyTorch, TensorFlow and JAX, uses CUDA and its libraries to run on NVIDIA GPUs.

An everyday picture: marking exam papers. One expert teacher can mark any paper but only one at a time. A hall of 5,000 volunteers, each given simple instructions and one page, can finish a mountain of identical pages in minutes. A CPU is the expert teacher: a few powerful cores for complex, varied work. A GPU is the hall of volunteers, and CUDA is the system for writing the instructions, handing out pages and collecting results.

Deep learning is mostly huge numbers of multiplications and additions on large arrays, exactly the kind of work that splits into many identical pieces. That is why CUDA became the foundation of modern AI computing, and why NVIDIA's software ecosystem matters as much as its hardware.

What it is

Most AI practitioners never write raw CUDA. They call frameworks that call these libraries. But understanding the model explains why GPUs behave as they do, and custom kernels are still written for performance-critical work such as FlashAttention.

Where it came from

Before CUDA, researchers used GPUs for general computation by disguising their problems as graphics, writing shaders that drew pixels. Academic projects such as Brook at Stanford, led by Ian Buck, showed that a cleaner programming model was possible. Buck joined NVIDIA, and the company announced CUDA in 2006 alongside the GeForce 8800, the first NVIDIA GPU with a unified architecture designed for general computing. The first public toolkit followed in 2007.

CUDA grew steadily: double precision, unified virtual addressing, libraries such as cuBLAS and cuFFT, and, in 2014, cuDNN for deep learning. The breakthrough moment for AI came in 2012, when the AlexNet image model was trained on NVIDIA GPUs using CUDA, showing how much faster deep networks could be trained. Later generations added Tensor Cores for matrix maths, mixed precision and support for new number formats. CUDA 13.0, released in August 2025, introduced a tile-based programming model, and NVIDIA has since added tile programming from Python.

How it works

Host and device

A CUDA program has host code that runs on the CPU and device code that runs on the GPU. The host allocates GPU memory, copies data, launches kernels and copies results back, or uses unified memory that moves pages automatically.

Threads, blocks and grids

A kernel launch creates a grid of thread blocks. Each block contains up to 1,024 threads, and each thread knows its position through built-in indices. Threads run in groups of 32, called warps, that execute the same instruction together.

__global__ void add(const float* a, const float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}
// launch: add<<<(n + 255) / 256, 256>>>(a, b, c, n);

Streaming multiprocessors

The GPU is made of streaming multiprocessors, each running many warps at once. When one warp waits for memory, another runs, which hides latency. This is why GPUs need many threads to perform well.

The memory hierarchy

Performance depends heavily on memory:

• Registers are private to each thread and fastest.
• Shared memory is a small, fast scratchpad shared by threads in a block.
• L2 cache is shared by the whole GPU.
• Global memory, the HBM or GDDR on the card, is large but slower.

Fast kernels load data from global memory once, reuse it from shared memory and registers, and access memory in coalesced patterns where neighbouring threads read neighbouring addresses.

Tensor Cores

Modern NVIDIA GPUs include Tensor Cores, units that multiply small matrix tiles in one step at reduced precisions such as FP16, BF16, FP8 and, on the newest generations, FP4. Libraries such as cuBLAS and cuDNN use them automatically for deep learning.

Streams and concurrency

Streams are queues of work. Operations in different streams can overlap, for example copying the next batch while computing the current one. CUDA Graphs record sequences of launches to replay them with less CPU overhead.

Key concepts and terms

A worked example

This scenario is illustrative. A data team computes similarity scores between 2 million product embeddings and 10,000 query embeddings, each of 768 dimensions, every night.

1. The job runs for 50 regional catalogues. On a 32-core CPU with an optimised library, each multiplication of 2,000,000 by 768 by 10,000 values, about 1.5 times 10 to the power 13 multiply-adds, takes around 30 seconds, so scoring alone takes about 25 minutes.
2. They move the arrays to a data-centre GPU and call cuBLAS through PyTorch in BF16. Each multiplication now takes well under a second, because Tensor Cores process matrix tiles in parallel.
3. Profiling with Nsight Systems shows the GPU is now idle most of the time, waiting for data loading from disk. They overlap loading and compute with separate streams.
4. A custom step that picks the top 100 matches per query runs slowly as a series of framework operations. A small fused kernel that keeps partial results in shared memory reduces that step from 40 to 6 seconds per catalogue.
5. The whole nightly job falls from more than 30 minutes to a few minutes.

Strengths and limitations

Strengths:

• Massive parallel performance for suitable workloads.
• The most mature GPU software ecosystem, with highly optimised libraries.
• Supported by every major AI framework.
• Excellent profiling and debugging tools.
• Long-term compatibility through PTX across GPU generations.

Limitations:

• Runs only on NVIDIA GPUs, creating vendor dependence.
• Writing fast kernels requires deep knowledge of memory and parallelism.
• Version matching between drivers, toolkits and frameworks can be painful.
• Workloads with heavy branching or little parallelism gain little.
• Data transfer between CPU and GPU can erase gains if not managed.

How it compares with alternatives

Practical guidance and best practices

1. Use libraries and frameworks first; write custom kernels only after profiling shows a need.
2. Profile before optimising, with Nsight Systems for the whole program and Nsight Compute for kernels.
3. Keep data on the GPU between steps to avoid repeated transfers.
4. Use mixed precision so Tensor Cores can be used.
5. Overlap data movement and compute with streams or asynchronous loaders.
6. Match driver, toolkit and framework versions using official compatibility tables.
7. Use containers with tested CUDA stacks to avoid environment drift.
8. Check memory access patterns for coalescing in custom kernels.

Common mistakes

• Copying data back to the CPU after every small operation.
• Launching many tiny kernels instead of fusing work.
• Installing a framework built for a different CUDA version than the driver supports.
• Measuring time without synchronising, since kernel launches are asynchronous.
• Ignoring out-of-bounds checks in kernels, which corrupt memory silently.
• Assuming the GPU is the bottleneck without checking data loading.

Frequently asked questions

Do I need to learn CUDA to do deep learning?

Usually not. Frameworks such as PyTorch handle CUDA for you. Knowing the basics helps with performance tuning, debugging memory errors and understanding hardware choices.

What is the difference between CUDA and cuDNN?

CUDA is the overall platform and programming model. cuDNN is one library built on CUDA, with optimised routines for neural network operations such as convolution and attention.

Can CUDA run on AMD or Intel GPUs?

Not natively. CUDA targets NVIDIA GPUs. AMD's ROCm and HIP offer a similar model, and tools exist to port CUDA code, but compatibility is not automatic.

What is compute capability?

It is a version number for each NVIDIA GPU architecture that tells the compiler which features and instructions are available, such as particular Tensor Core formats.

Why is GPU memory bandwidth so important?

Many AI operations, especially generating text with large models, read huge amounts of data for relatively little maths. Their speed is limited by how fast memory can be read, not by compute.

Glossary

• Host: the CPU and its memory.
• Device: the GPU and its memory.
• HBM: high-bandwidth memory stacked next to the GPU chip.
• Coalescing: combining neighbouring threads' memory accesses into fewer transactions.
• Kernel fusion: combining several operations into one kernel to save memory traffic.
• CUDA Graph: a recorded set of GPU operations replayed with low overhead.
• Mixed precision: using lower precision for most maths while keeping key values in higher precision.

Summary

CUDA is NVIDIA's platform for running parallel programs on GPUs, organising work into kernels, threads, warps and blocks, exposing a memory hierarchy from registers to HBM, and providing libraries such as cuBLAS, cuDNN and NCCL. Launched in 2006 and central to deep learning since the early 2010s, it now includes tile-based programming introduced with CUDA 13 in 2025. Most practitioners use it through frameworks, but understanding its model explains GPU performance, and its maturity is a major reason NVIDIA dominates AI computing.

Related

It belongs to Hardware in my stack, under MLOps and AI infrastructure. Others in the same group: