NVIDIA H200

Overview

The NVIDIA H200 is a data-centre GPU based on the same Hopper architecture as the H100, with one major change: it has much more and much faster memory. Each H200 carries 141 GB of HBM3e memory with about 4.8 TB/s of bandwidth, compared with 80 GB and about 3.35 TB/s on the H100 SXM. The compute units, Tensor Cores and Transformer Engine are essentially the same, so the H200's advantage shows up in workloads limited by memory size or speed, above all large language model inference with long contexts.

An everyday picture: two delivery vans with the same engine. One has a small cargo area and a narrow loading door; the other has a much bigger cargo area and a wider door. On short trips with small parcels they perform the same. When the job is moving lots of bulky goods, the bigger van finishes far sooner, because it carries more per trip and loads faster. The H200 is the bigger van: same engine as the H100, larger and faster cargo space.

Because the H200 fits the same server designs as the H100, organisations could upgrade without redesigning systems or software. That made it a popular step between the H100 and the Blackwell-generation B200.

What it is

NVIDIA publishes these figures for the two H200 versions. Peak Tensor Core figures are quoted with sparsity; dense figures are half.

The SXM version is used in HGX H200 boards with eight GPUs connected by NVSwitch, giving about 1.1 TB of GPU memory in one server. The NVL version is a PCIe card for standard, air-cooled enterprise servers, with NVLink bridges linking small groups of cards.

Where it came from

NVIDIA announced the H200 in November 2023 at the SC23 supercomputing conference. It was the first GPU to use HBM3e, a faster generation of high-bandwidth memory. Systems and cloud instances became available during 2024 from server makers and major cloud providers.

The timing reflected a shift in AI workloads. Training remained important, but serving large language models to many users had become a huge share of GPU demand, and inference is often limited by memory rather than compute. The H200 addressed that without waiting for a new architecture. The H200 NVL followed later for customers who wanted Hopper with large memory in conventional PCIe servers. NVIDIA's Blackwell architecture, announced in 2024, succeeded Hopper as the flagship.

How it works

Same compute, bigger memory

The H200 uses Hopper's streaming multiprocessors, fourth-generation Tensor Cores, the FP8 Transformer Engine and the same programming model through CUDA. Software written for the H100 runs unchanged. The difference is the memory system: six stacks of HBM3e instead of five of HBM3, for 141 GB in total, with faster signalling that raises bandwidth to 4.8 TB/s.

Why bandwidth matters for LLM inference

When a language model generates text, each new token requires reading all the active weights and the KV cache for every request in the batch. The maths per byte read is low, so the GPU spends much of its time waiting for memory. A rough lower bound for the time of one decode step is the bytes read divided by bandwidth. Raising bandwidth from 3.35 to 4.8 TB/s, about 43 per cent, can shorten each step proportionally when memory is the bottleneck.

Why capacity matters

Capacity determines what fits on each GPU:

• Larger models can run on fewer GPUs, cutting communication between them.
• More KV cache allows more concurrent users or longer contexts per GPU.
• Bigger batches improve throughput, because the weights read in each step are shared across more requests.

For example, a 70-billion-parameter model in FP8 needs about 70 GB for weights. On an H100 that leaves almost nothing for cache; on an H200 it leaves around 70 GB.

Training

For training, the extra memory allows larger batches, longer sequences or less activation recomputation, and can reduce the number of GPUs needed for a given model. Gains are usually smaller than for inference, because training is more compute-bound.

Scaling

Multi-GPU and multi-node scaling works exactly as with the H100: NVLink and NVSwitch inside servers, InfiniBand or Ethernet between them, and NCCL for collectives.

Key concepts and terms

A worked example

This scenario is illustrative, and the numbers are rough estimates rather than benchmarks. A company serves an open 70-billion-parameter model with 32,000-token contexts to an internal assistant.

1. In FP8 the weights take about 70 GB. For this model, assume each request's KV cache at full context needs roughly 5 GB in FP8.
2. On two H100 SXM GPUs with tensor parallelism, there are 160 GB in total. After weights and overhead, about 80 GB remain for cache, so roughly 16 full-length conversations fit at once.
3. On two H200 SXM GPUs, there are 282 GB. About 200 GB remain for cache, so roughly 40 full-length conversations fit.
4. Each decode step must read weights and cache. With higher bandwidth, steps on the H200 pair are faster, and with larger batches each step serves more users.
5. The team estimates that the H200 pair serves about two to three times as many long-context users as the H100 pair, so they compare that gain with the difference in rental price before deciding.

Strengths and limitations

Strengths:

• 141 GB of memory, about 76 per cent more than the H100.
• 4.8 TB/s of bandwidth, about 43 per cent more than the H100 SXM.
• Large gains for memory-bound inference and long contexts.
• Drop-in compatible with H100 software and server designs.
• PCIe NVL version for conventional air-cooled servers.

Limitations:

• Peak compute is the same as the H100, so compute-bound jobs gain little.
• No FP4 support, unlike Blackwell GPUs.
• High power draw, up to 700 W for the SXM version.
• Superseded as the flagship by the B200 and other Blackwell parts.
• Pricing must be weighed against the size of the actual gain for your workload.

How it compares with alternatives

Practical guidance and best practices

1. Profile first: if your workload is memory-bound, the H200 helps; if compute-bound, it may not.
2. Use the extra memory for larger batches or longer contexts, not idle headroom.
3. Combine with FP8 weights and an efficient KV cache to multiply the gain.
4. Reduce tensor parallelism where the model now fits on fewer GPUs, cutting communication.
5. Compare cost per generated token, not just hourly price.
6. Choose the NVL version only when standard PCIe servers and air cooling are required.
7. Set power limits deliberately if racks are power-constrained, and measure the effect.
8. Keep the software stack current; inference engines improve memory use over time.

Common mistakes

• Expecting faster training on compute-bound jobs because the H200 is newer.
• Comparing H200 sparse figures with dense figures from another GPU.
• Leaving batch sizes unchanged and wasting the extra memory.
• Assuming the NVL card performs like the SXM version.
• Forgetting KV cache in memory planning, even with 141 GB.

Frequently asked questions

What is the main difference between the H100 and H200?

Memory. The H200 has 141 GB of HBM3e at 4.8 TB/s, while the H100 SXM has 80 GB of HBM3 at about 3.35 TB/s. Their compute capabilities are essentially the same.

Is the H200 faster than the H100?

For memory-bound work such as large language model inference, often yes, sometimes substantially. For compute-bound work, the difference is small.

Can I run H100 software on an H200?

Yes. Both use the Hopper architecture and the same CUDA features, so frameworks, kernels and engines run without changes.

What is the H200 NVL?

It is a PCIe card version of the H200 for air-cooled servers, with up to 600 W power, the same 141 GB memory and NVLink bridges that connect two or four cards.

Should I choose an H200 or a B200?

The B200 offers more compute, more memory, FP4 and faster NVLink, but it is newer and may cost more or need different infrastructure. The H200 is simpler to adopt in existing Hopper environments. Compare cost per unit of work for your own models.

Glossary

• HBM stack: a vertical pile of memory chips connected to the GPU.
• Bytes per token: how much data must be read to generate one token.
• Batch size: the number of requests processed together in one step.
• Long context: inputs of many thousands of tokens.
• NVLink bridge: a connector linking PCIe GPUs directly.
• Rack power budget: the maximum power available to a rack of servers.
• Drop-in upgrade: a replacement that fits existing systems without redesign.

Summary

The NVIDIA H200 keeps the Hopper compute of the H100 but upgrades memory to 141 GB of HBM3e at 4.8 TB/s, which makes it markedly better for memory-bound jobs such as serving large language models with long contexts and many users. Announced in November 2023 and shipped from 2024 in SXM and PCIe NVL versions, it runs H100 software unchanged. It is the right choice when memory, not arithmetic, is the bottleneck, and it should be compared with the Blackwell B200 on cost per unit of work.

Related

It belongs to Hardware in my stack, under MLOps and AI infrastructure. Others in the same group: