# NVIDIA H100

The NVIDIA H100 is a Hopper-architecture data-centre GPU with a Transformer Engine and FP8 support, widely used to train and serve large AI models.

## Overview

The NVIDIA H100 is a data-centre GPU built on NVIDIA's Hopper architecture. It was designed for training and serving large AI models and for high-performance computing. Its headline features are fourth-generation Tensor Cores, a Transformer Engine that uses the 8-bit FP8 number format to speed up transformer models, fast HBM3 memory and fourth-generation NVLink for connecting GPUs. From 2023 onwards it became one of the most widely used chips for building and running large language models.

An everyday picture: a professional kitchen upgraded for one popular dish. If most orders are pasta, the restaurant installs a dedicated pasta station, faster boilers and a tool that portions ingredients precisely without waste. The kitchen still cooks everything, but the most common dish gets much faster. The H100 is a general GPU with special equipment for the dominant AI workload of its time, the transformer, and the Transformer Engine is the portioning tool that uses smaller numbers where precision allows.

The H100 sits in a family. The [H200](/tools/nvidia-h200) uses the same Hopper compute with more and faster memory, and the [B200](/tools/nvidia-b200) is the next generation, built on the Blackwell architecture.

## What it is

The H100 comes in several form factors. The figures below are for the H100 SXM, the version used in most large AI servers, as published by NVIDIA. Peak Tensor Core figures for low precisions are often quoted "with sparsity", which is twice the dense figure.

| Specification | H100 SXM |
|---|---|
| Architecture | Hopper (GH100 chip) |
| Streaming multiprocessors | 132 |
| Memory | 80 GB HBM3 |
| Memory bandwidth | About 3.35 TB/s |
| FP8 Tensor Core | About 1,979 TFLOPS dense, 3,958 with sparsity |
| BF16 and FP16 Tensor Core | About 989 TFLOPS dense, 1,979 with sparsity |
| TF32 Tensor Core | About 495 TFLOPS dense |
| FP64 Tensor Core | About 67 TFLOPS |
| NVLink | Fourth generation, 900 GB/s total per GPU |
| Host interface | PCIe Gen5 |
| Maximum power | Up to 700 W |
| Multi-Instance GPU | Up to 7 isolated instances |

Other versions include a PCIe card with lower power and HBM2e memory of about 2 TB/s bandwidth, and the H100 NVL, a pair-based version for inference with 94 GB per GPU. Eight H100 SXM GPUs connected by NVSwitch form an HGX H100 board, the building block of most H100 servers.

## Where it came from

NVIDIA announced the Hopper architecture and the H100 at its GTC conference in March 2022, naming the architecture after the computing pioneer Grace Hopper. The GH100 chip has about 80 billion transistors and is made on a TSMC 4N process. It succeeded the Ampere-based A100, which had been the standard AI accelerator since 2020.

H100 systems began shipping in late 2022. Demand rose sharply in 2023 as companies raced to train and serve large language models, and H100s were in short supply for much of that year. Cloud providers offered H100 instances, and many of the largest AI clusters of 2023 and 2024 were built from tens of thousands of them. The H200, announced in November 2023, kept Hopper's compute and upgraded the memory; Blackwell followed in 2024.

## How it works

### Tensor Cores and precision

Most AI computation is matrix multiplication. Tensor Cores multiply small matrix tiles in one operation. Hopper's fourth-generation Tensor Cores support FP64, TF32, BF16, FP16, INT8 and, new in this generation, FP8. Halving the number of bits roughly doubles the peak operations per second and halves memory traffic.

### The Transformer Engine

FP8 has a narrow range, so using it naively can lose accuracy. The Transformer Engine, a combination of hardware and NVIDIA software, tracks the range of values in each layer and chooses scaling factors so that matrix multiplications can run in FP8 while sensitive parts stay in higher precision. Frameworks access it through NVIDIA's Transformer Engine library and through optimised inference engines.

### Memory

Each H100 SXM has 80 GB of HBM3 stacked beside the GPU die. For many large language model tasks, especially generating tokens, speed depends on how fast weights and the [KV cache](/tools/kv-cache) can be read, so memory bandwidth matters as much as peak compute. Memory capacity decides how large a model, batch or context fits on one GPU.

### Other Hopper features

- **Thread block clusters and distributed shared memory** let groups of thread blocks cooperate across multiprocessors.
- **The Tensor Memory Accelerator** moves blocks of data between global and shared memory efficiently, which kernels such as [FlashAttention](/tools/flashattention) use.
- **DPX instructions** accelerate dynamic programming algorithms used in genomics and route optimisation.
- **Confidential computing** support protects data and models while in use.

### Scaling

Inside an HGX server, eight H100s connect through NVSwitch so every GPU can reach every other at full NVLink bandwidth. Across servers, high-speed networks such as InfiniBand link GPUs, with [NCCL](/tools/nccl) handling collective communication. Software such as [CUDA](/tools/cuda), cuDNN, [TensorRT](/tools/tensorrt) and engines such as [vLLM](/tools/vllm) exploit these features.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Hopper | NVIDIA's GPU architecture generation that includes the H100 |
| SXM | A socketed module form factor for high-power GPUs in servers |
| HBM3 | Third-generation high-bandwidth memory |
| Tensor Core | A unit for fast matrix multiplication |
| FP8 | An 8-bit floating-point format, in E4M3 and E5M2 variants |
| Transformer Engine | Hardware and software that manage FP8 scaling for transformers |
| Sparsity | A feature that skips structured zero weights to double peak throughput |
| NVLink | High-bandwidth GPU-to-GPU connection |
| NVSwitch | A switch that connects many GPUs by NVLink |
| MIG | Multi-Instance GPU, splitting one GPU into isolated parts |
| HGX | NVIDIA's server board with several connected GPUs |
| TDP | Thermal design power, the maximum sustained power draw |

## A worked example

This scenario is illustrative, and the numbers are rough estimates rather than benchmarks. A company wants to serve an open 70-billion-parameter language model.

1. In BF16, the weights need about 140 GB, more than one H100's 80 GB. In FP8, they need about 70 GB, which fits on one GPU but leaves little room for KV cache.
2. They choose two H100 SXM GPUs with tensor parallelism and FP8 weights. That leaves about 90 GB across the pair for KV cache and activations.
3. Each generated token requires reading the weights once. At about 3.35 TB/s per GPU, reading 35 GB per GPU sets a lower bound of roughly 10 milliseconds per decode step, before communication and overhead. Batching many users into each step spreads that cost.
4. Using an engine with continuous batching, they serve several dozen concurrent chats per pair while meeting their target time per token.
5. When they later need longer contexts, they compare adding more H100s with moving to H200s, whose larger memory holds more KV cache per GPU.

## Strengths and limitations

**Strengths:**

- Large performance step over the A100, especially with FP8.
- Transformer Engine makes reduced precision practical for transformers.
- Mature CUDA software stack and broad framework support.
- Fast NVLink and NVSwitch for multi-GPU servers.
- Widely available from cloud providers and server makers.

**Limitations:**

- 80 GB of memory is tight for the largest models and long contexts.
- High power draw and cooling requirements.
- Expensive to buy and rent, though prices have fallen as newer GPUs arrived.
- FP8 needs care to avoid accuracy loss.
- Superseded in peak performance by Blackwell-generation GPUs.

## How it compares with alternatives

| GPU or chip | Memory | Notes |
|---|---|---|
| NVIDIA A100 | 40 or 80 GB HBM2e | Ampere predecessor, no FP8 |
| NVIDIA H100 SXM | 80 GB HBM3 | Hopper, FP8 Transformer Engine |
| [NVIDIA H200](/tools/nvidia-h200) | 141 GB HBM3e | Same compute, more and faster memory |
| [NVIDIA B200](/tools/nvidia-b200) | Up to about 180 to 192 GB HBM3e, depending on configuration | Blackwell, adds FP4, faster NVLink |
| [Cloud TPU](/tools/cloud-tpu) | Varies by generation | Google's custom accelerators |

## Practical guidance and best practices

1. Estimate memory first: weights, KV cache, activations and overhead, before choosing GPU counts.
2. Use BF16 for training and consider FP8 with the Transformer Engine where accuracy holds.
3. Keep tensor parallelism inside one NVLink-connected server.
4. Use optimised engines and libraries rather than plain framework code for serving.
5. Use MIG to share a GPU among small workloads safely.
6. Monitor utilisation, memory and power with tools such as NVIDIA DCGM.
7. Plan for 700 W per GPU in power and cooling budgets for SXM systems.
8. Compare cost per token or per training step, not just hourly price, across GPU types.

## Common mistakes

- Comparing sparse peak figures with dense figures from another chip.
- Assuming a model fits because the weights fit, forgetting the KV cache.
- Running tensor parallelism across servers over slower networks.
- Switching to FP8 without validating accuracy.
- Using the PCIe version when the workload needs NVLink bandwidth.
- Leaving GPUs underused with small batches.

## Frequently asked questions

### What is the difference between H100 SXM and H100 PCIe?

The SXM version is a module for servers with up to 700 W of power, HBM3 memory and full NVLink connectivity. The PCIe version is a standard card with lower power, HBM2e memory, lower bandwidth and lower peak performance, suited to ordinary servers.

### What does FP8 mean for AI?

FP8 is an 8-bit number format. Using it for matrix maths roughly doubles throughput compared with 16-bit formats and halves memory traffic. The Transformer Engine manages scaling to keep accuracy acceptable.

### How much memory does an H100 have?

The H100 SXM and PCIe versions have 80 GB. The H100 NVL has 94 GB per GPU. The related H200 has 141 GB.

### Is the H100 still a good choice?

For many workloads, yes. It is widely available, well supported and cheaper than when it launched. Blackwell GPUs offer more performance and memory, so the right choice depends on price, availability and workload.

### What is MIG?

Multi-Instance GPU splits one H100 into up to seven isolated instances, each with its own memory and compute, so several small jobs or users can share one GPU safely.

## Glossary

- Die: the piece of silicon that holds the GPU circuits.
- Dense throughput: peak speed without using sparsity.
- Decode step: generating one token for each request in a batch.
- Tensor parallelism: splitting each layer across several GPUs.
- DCGM: NVIDIA Data Center GPU Manager, for monitoring GPUs.
- Confidential computing: protecting data while it is being processed.
- TSMC 4N: the manufacturing process used for the GH100 chip.

## Summary

The NVIDIA H100 is a Hopper-architecture data-centre GPU, announced in 2022, with 132 multiprocessors, 80 GB of HBM3 at about 3.35 TB/s in its SXM form, fourth-generation Tensor Cores with FP8, a Transformer Engine and 900 GB/s NVLink. It became the workhorse of large-scale AI in 2023 and 2024 and remains widely used. Choosing it well means planning memory for weights and KV cache, keeping heavy communication inside NVLink domains and comparing cost per unit of work against the H200 and Blackwell-generation alternatives.