MLPerf Training v6.1 and the first Endpoints results, and how to read AI hardware claims
MLCommons published MLPerf Training v6.1 on 16 November 2026, and the same day NVIDIA published its first MLPerf Endpoints v1.0 agentic serving numbers. Here is what each benchmark measures, and how to read scaling and per-watt claims without being misled.
- Reading time
- 17 min
- Sections
- 7
- Code examples
- 1
- Tables
- 2
- Sources
- 12
- Published
- 17 Nov 2026
On 16 November 2026, MLCommons published the results of MLPerf Training v6.1. MLCommons calls it the largest training round so far, with 250 results from 24 organizations. It is also the first round that times reinforcement learning post-training, not just pretraining and fine-tuning. On the same day, NVIDIA published its first numbers on MLPerf Endpoints v1.0, a newer benchmark that tests AI serving through a live API, including agentic coding workloads.
These benchmarks are the industry's shared ruler for AI hardware and software. But the headline figures that reach the press are usually chosen by the company that ran them. This article explains what each benchmark measures, how to read scaling and per-watt claims, and why one footnote about 32 versus 48 accelerators matters more than the headline above it.
What happened
MLCommons publishes MLPerf Training v6.1 (16 November 2026)
MLCommons, the nonprofit consortium that runs MLPerf, released the v6.1 training results. Its press release reports 153 systems, 250 results and 24 submitting organizations, including first-time submitter Crusoe. Fifty-seven systems, or 37%, ran in the cloud, up from 29% in v6.0. Twelve systems used 1,000 or more accelerators, and the largest used 10,240.
The round added two benchmarks. DLRMv4-HSTU replaces the older DLRM-dcnv2 recommender. Qwen3.5-397B is a reinforcement learning post-training workload that teaches a model to fix real software bugs with tools. MLCommons says this round includes the first results on NVIDIA's Rubin-generation GPUs and on Google's TPUv7, called Ironwood.
Working group co-chair Shriya Rishab said: "Post-training has added a new level of complexity to the architecture of AI training."
NVIDIA publishes training claims for Vera Rubin NVL72 (16 November 2026)
NVIDIA says its Vera Rubin NVL72 rack posted the fastest time to train on six of the seven benchmarks in the closed division. It claims up to 2x the per-GPU performance of its own GB300 NVL72. It also says NVIDIA platforms were the only ones with results on all seven benchmarks. These are company claims built on MLCommons result IDs.
The largest runs came from two submitters. NVIDIA ran DeepSeek-V3 on 10,240 GB300 GPUs and reports 1.38 minutes, the fastest absolute time. Microsoft Azure matched that scale with 10,240 GB200 GPUs. TechArena reports that Google ran the same benchmark on 8,192 Ironwood chips.
First MLPerf Endpoints v1.0 agentic results go public (16 November 2026)
MLCommons opened Endpoints v1.0 for submissions on 13 October 2026 and said the first results were expected in November. Endpoints uses rolling submissions, so results appear when they are ready, not on one release day. On 16 November, NVIDIA published blog posts on its Endpoints v1.0 entries. The headline claim is more than 10x the throughput per kilowatt of GB300 NVL72 on the Kimi K3 agentic workload. NVIDIA's own footnote marks the Vera Rubin result as provisional, pending MLCommons verification.
How MLPerf Training measures progress
The metric is time to a quality target
MLPerf Training measures one thing: the wall-clock time a system needs to train a given model on a given dataset until it reaches a given quality target. A system that processes data fast but never reaches the target gets no score.
Think of it as a marathon timed at the finish line. Only the time across the line counts, and only if the runner finished the full course.
Each benchmark is run several times. MLCommons drops the fastest and slowest runs and averages the rest. It says the typical run-to-run variance is about plus or minus 2.5% for imaging benchmarks and about plus or minus 5% for the others. That matters: a 3% lead on a language benchmark sits inside normal noise.
The v6.1 benchmark list
MLCommons lists these current training benchmarks on its benchmark page. Seven of them were run in v6.1, according to NVIDIA and TechArena.
| Task | Model | Dataset | Quality target |
|---|---|---|---|
| LLM pretraining | DeepSeek-V3 (671B MoE) | C4 | 3.6 log perplexity |
| LLM pretraining | GPT-OSS 20B | C4 | 3.34 log perplexity |
| LLM pretraining | Llama 3.1 8B | C4 | 3.3 log perplexity |
| LLM fine-tuning (LoRA) | Llama 2 70B | SCROLLS GovReport | 0.925 cross-entropy loss |
| LLM RL post-training | Qwen3.5-397B-A17B | R2E-Gym subset | 0.69 pass@4 |
| Image generation | FLUX.1 | cc12m | 0.586 validation loss |
| Recommendation | DLRMv4-HSTU | Yambda | 0.75 AUC |
Three of these use a Mixture-of-Experts design, where only part of the model runs for each token. DeepSeek-V3 has 671 billion parameters in total. Qwen3.5-397B has 397 billion in total but only 17 billion active per token.
The new RL post-training benchmark
In the Qwen3.5-397B benchmark, a coding agent tries to fix bugs in real Python projects such as NumPy, Pandas and Sympy, inside a container. Each attempt is called a rollout. Each rollout gets a simple pass or fail reward from hidden tests.
The training method is GRPO, short for Group Relative Policy Optimization, a form of reinforcement learning. The system samples several rollouts for the same problem, compares their rewards inside that group, and nudges the model toward the better ones. The setup allows 16 generations per prompt, 30 agent turns and a 65,536-token context.
The quality target is pass@4 of 0.69. That means the model must solve 69% of validation problems at least once in four tries. MLCommons says it chose pass@4 because it had the lowest variance of the metrics it tested.
This benchmark is harder to run than pretraining. MLCommons notes that agentic RL is far less deterministic: tool calls, timing and numerical differences add up over many turns. The generator and the trainer must also keep pace with each other, or the bottleneck just moves. Multi-turn rollouts use a lot of KV cache memory, so systems must balance keeping that cache against recomputing it.
Closed versus open divisions and availability labels
The closed division requires the same model as the reference implementation, so it compares hardware and software "apples to apples." The open division allows a different model or retraining, to encourage new ideas.
Each result also carries an availability label:
- Available: built only from parts you can buy or rent in the cloud today.
- Preview: must be submittable as Available in the next round.
- RDI: research, development or internal hardware or software.
TechArena reports that every v6.1 training result was in the closed division's Available category.
How MLPerf Endpoints measures serving
The system under test is a URL
MLCommons introduced MLPerf Endpoints in March 2026. The older MLPerf Inference design tied the test harness closely to the hardware. Endpoints instead sends requests to a serving endpoint over standard interfaces such as HTTP or gRPC. Because the system under test is a URL, a bare-metal server, a cloud service and a managed API can all be tested the same way. Our guide to the LLM inference serving stack shows what sits behind such an endpoint.
One result is a curve, not a number
Serving speed has two sides that pull against each other. System throughput is the total tokens per second across all users. Interactivity is the tokens per second each user sees. Add more concurrent users and total throughput rises, but each user waits longer.
Picture a restaurant kitchen. It can serve more meals per hour if it lets every table wait an hour. A fair test asks how many meals per hour the kitchen serves while every table still gets food within, say, 15 minutes.
Endpoints captures this trade-off by sweeping load. MLCommons says a result is a Pareto curve with 7 to 32 points, each run for at least 600 seconds, on the same model, endpoint and software stack. The curve is drawn as a step function, so it shows only measured points and never interpolated ones.
Endpoints also reports time to first token at the 90th percentile (TTFT P90) and the concurrency used. Version 1.0 adds mandatory provisioned power measurement, so results can be shown as throughput per watt.
Agentic workloads and correctness checks
Version 1.0 adds agentic coding workloads. MLCommons named DeepSeek v4 Pro, Kimi K3 and Qwen 3.6 when it opened submissions. The results explorer lists deepseek-v4_1-flash, kimi-k3 and Qwen3.6 35B-A3B under the agentic type.
These workloads are heavy. Coding sessions build up history over many turns, and NVIDIA's chart shows median context rising from about 30,000 tokens at the first response to about 200,000 tokens by response 300. Enterprise workflows reuse shared instructions and tool definitions, which rewards prompt caching.
Speed means nothing if the answers are wrong. MLCommons says accuracy is validated for each distinct configuration. NVIDIA says that during performance runs, generated tool calls and workflow intent are checked against reference values, and SWE-bench Verified is run separately to test end-to-end task solving.
How to read scaling, per-watt and status claims
Scaling efficiency
With four times as many GPUs, the ideal is a training time four times shorter. Scaling efficiency compares the real speed-up with that ideal.
NVIDIA says Vera Rubin NVL72 scaled DeepSeek-V3 from 256 to 1,024 GPUs at about 97% efficiency, cutting training time from about 16 minutes to about four. Its technical blog gives 4.14 minutes at 1,024 GPUs. A quick check: 16 divided by 4 is 4.0 minutes, and 4.0 divided by 4.14 is about 0.97. The claim is consistent with its own numbers.
High scaling efficiency is hard because chips spend more time talking to each other as the cluster grows. Techniques such as data parallelism and tensor parallelism split the work, but every split adds communication.
NVIDIA also gives a generation-to-generation view at 1,024 GPUs on DeepSeek-V3: 12.36 minutes on GB200, 8.03 minutes on GB300 and 4.14 minutes on Vera Rubin. These are NVIDIA's figures.
Per-GPU claims need the same GPU count
NVIDIA's "up to 2x per GPU" claim compares Vera Rubin and GB300 at the same GPU count on each benchmark. Its table shows the ratio varies: 2x on DeepSeek-V3 at 1,024 GPUs, but 1.4x on the Qwen3.5 GRPO run at 64 GPUs and 1.4x on DLRMv4 at 128 GPUs. "Up to" means the best case.
Nebius reports its 64-GPU Vera Rubin system trained the GRPO benchmark in 24.82 minutes, against 37.01 minutes for its 64-GPU GB300 system. Its "first place" claim covers only same-GPU-type submissions at 64 GPUs, not all of MLPerf.
Verified, provisional and unverified
This is the most important label to check. On the Endpoints explorer, a provisional result is pending MLCommons verification and peer review and may change. An official result is verified and peer reviewed. When we checked the explorer's agentic page, eight of the ten DeepSeek-V4.1-Flash entries were provisional and two were official.
Some numbers are neither. NVIDIA's training blog includes reruns on newer software that it labels as not verified by MLCommons. For example, it shows Vera Rubin training DLRMv4 in 1.628 minutes on a rerun, against 2.829 minutes in its official result. Those reruns may be real progress, but they are not MLPerf results. Treat them like a student grading their own homework.
The Kimi K3 footnote
NVIDIA's headline Endpoints claim is more than 10x the throughput per kilowatt of GB300 NVL72 on Kimi K3, at a shared minimum of 50 tokens per second per user. The footnote shows how the two runs differ.
| Item (NVIDIA footnote) | Vera Rubin NVL72 | GB300 NVL72 |
|---|---|---|
| Status | Provisional | Verified |
| Accelerators | 32 | 48 |
| Concurrent queries | 1,024 | 48 |
| TTFT P90 | 4,673.9 ms | 2,305.1 ms |
| System throughput | 41,414.2 tokens/s | 2,210.4 tokens/s |
| Interactivity per user | 50.4 tokens/s | 56.2 tokens/s |
| Power | 100.89 kW | 60 kW |
Our own arithmetic from these figures gives about 410 tokens per second per kilowatt for Vera Rubin and about 37 for GB300, a ratio near 11x. So the headline follows from the footnote. But look at what the comparison is.
Each platform is shown at its best throughput per kilowatt among runs that clear 50 tokens per second per user. Vera Rubin cleared the floor while serving 1,024 users at once. The GB300 point chosen served only 48. NVIDIA's own caveat says comparisons use each platform's best point above the floor, not identical configurations.
This answers a fair buyer question: how much work per kilowatt can each system do while users still get at least 50 tokens per second? But it is not the same as "the same rack does ten times more work." The first-token wait on the Vera Rubin run was also about twice as long. If your users care most about the first token, the ranking may differ.
What it means in practice
Training benchmarks now cover the work most teams actually do. Few companies pretrain a 671-billion-parameter model. Many fine-tune and run RL post-training. The Qwen3.5 GRPO benchmark gives the first shared measurement of that full loop, including rollouts, tool calls and weight updates. It drew 29 submissions in its first round, according to TechArena.
Power is becoming a first-class metric. Endpoints v1.0 makes provisioned power mandatory. Training results rarely include energy; TechArena reports one measured-energy result in v6.1. For teams limited by data center power, as discussed in our piece on AI infrastructure bottlenecks, throughput per kilowatt at a fixed user speed is often the number that decides capacity.
Public buyers are using MLPerf directly. MLCommons says the EU's AI Gigafactories tender names MLPerf as the standard for measuring technical performance, scoring proposals on Training v5.1 and Inference v5.1. That raises the stakes for careful reading.
Software matters as much as silicon. TechArena reports that at eight accelerators, best times improved 2.03x on Llama 2 70B LoRA and 1.97x on GPT-OSS-20B. According to TechArena, working group co-chair Pavan Yalamanchili credited such gains partly to more efficient kernels and better overlap of communication with computation. The hardware you already own may get faster with the next software release.
What to do now
- Start from the MLCommons results, not the press release. Use results.mlcommons.org and endpoints.mlcommons.org to see every submitter, not just the one telling the story.
- Check the status label first. Separate verified, provisional and unverified numbers before you compare anything.
- Match the accelerator count. Compare per-GPU or per-chip claims only at the same scale. A 1,024-GPU result says little about an 8-GPU node.
- Read the footnotes for concurrency and floors. For serving claims, find the concurrency, the interactivity floor and the TTFT for each side.
- Pick the benchmark closest to your workload. If you fine-tune, look at Llama 2 70B LoRA. If you run RL on agents, look at Qwen3.5 GRPO. If you serve long agent sessions, look at the Endpoints agentic results.
- Set your own service floor. Decide the minimum tokens per second per user and the maximum first-token wait your product needs, then read Endpoints curves at that point. Our guide to cost and latency budgets can help.
- Ask vendors for their curve, not their peak. Request the full Pareto curve and the provisioned power for the configuration they plan to sell you.
- Run your own test before you sign. Your own prompts, context lengths and tool calls decide the winner. Our article on why a score is only as good as its report covers what a trustworthy report should include.
Risks and open questions
Provisional results can change. The Vera Rubin Endpoints numbers are pending verification and peer review. Until they are verified, treat the 10x figure as a company claim.
Best-point comparisons can hide trade-offs. Comparing each system at its own best point above a floor is fair for one question, but it can mask longer first-token waits, different concurrency and different hardware sizes.
Coverage is uneven. In this training round, NVIDIA says only its platforms had results on all seven benchmarks. On the Endpoints agentic page we read, entries came from only three submitters: Krai, NVIDIA and Google. A missing result is simply no data.
Power figures are still settling. MLCommons's Endpoints page says normalization rules are still under development. Explorer entries mark some power values as MLCommons estimates. Compare per-watt numbers only when you know how the power was measured.
Watch next: whether the provisional Vera Rubin Endpoints results become verified unchanged, whether AMD and Google post agentic Endpoints results at rack scale, and how many submitters return to the RL post-training benchmark in the next training round.
Frequently asked questions
What does MLPerf Training actually measure?
It measures the wall-clock time a system needs to train a fixed model on a fixed dataset until it reaches a fixed quality target. MLCommons runs each benchmark several times, drops the fastest and slowest runs, and averages the rest.
What is new in MLPerf Training v6.1?
Two benchmarks are new. Qwen3.5-397B GRPO is the first reinforcement learning post-training benchmark, where a coding agent learns to fix real software bugs. DLRMv4-HSTU replaces the older DLRM-dcnv2 recommender. DeepSeek-V3 pretraining is also part of the current suite.
How is MLPerf Endpoints different from MLPerf Inference?
Endpoints tests a live API endpoint over HTTP or gRPC, so the system under test is a URL. It reports a curve of throughput against per-user speed across many load levels, rather than a single figure. It also uses rolling submissions and, in v1.0, mandatory provisioned power measurement.
What does a provisional MLPerf result mean?
A provisional result is pending MLCommons verification and peer review, and it may change. A verified or official result has passed that review. Unverified numbers, such as vendor reruns on newer software, are not MLPerf results at all.
Is the 10x Kimi K3 claim wrong?
The arithmetic in NVIDIA's footnote does support a ratio near 10x in throughput per kilowatt at a 50 tokens per second per user floor. But the two runs used different accelerator counts and very different concurrency, and the Vera Rubin result is provisional. It answers one fair buyer question, not every question.
Summary
MLPerf Training v6.1, published on 16 November 2026, extended the training suite into reinforcement learning post-training with a Qwen3.5-397B GRPO coding-agent benchmark and drew 250 results from 24 organizations. The same day, the first MLPerf Endpoints v1.0 agentic serving results went public, measuring throughput, per-user speed and provisioned power through live API endpoints. NVIDIA claims up to 2x per-GPU training gains and more than 10x throughput per kilowatt for Vera Rubin NVL72 over GB300 NVL72, but the serving result is provisional and compares runs at different scales and concurrency. Check the status label, accelerator count, speed floor and footnotes, then test on your own workload; our MLOps and AI infrastructure hub covers the wider picture.
Sources
- MLCommons: MLCommons Releases MLPerf Training v6.1 Results, Spotlighting Recent Advances in AI Training
- MLCommons: MLPerf Training benchmark page
- MLCommons: MLPerf Training v6.1, First LLM Post-Training Benchmark
- MLCommons: MLPerf Endpoints benchmark page
- MLCommons: MLPerf in the EU AI Gigafactories Tender, Endpoints v1.0 Open
- MLCommons: MLPerf Endpoints, a new approach to generative AI benchmarking
- MLCommons: MLPerf Endpoints results explorer
- NVIDIA Blog: NVIDIA Sweeps MLPerf Training and Endpoints as Vera Rubin Debuts With Leading Performance
- NVIDIA Technical Blog: NVIDIA Vera Rubin NVL72 Achieves up to 2x Blackwell Ultra Performance in MLPerf Training v6.1
- NVIDIA Technical Blog: NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0
- TechArena: In Record Round, MLPerf Training 6.1 Reaches Into Post-Training
- Nebius: MLPerf Training v6.1, Nebius ranks 1st on NVIDIA Vera Rubin at rack scale
Continue learning
Cite this article
Kelahmet, Ö. C. (2026, November 17). MLPerf Training v6.1 and the first Endpoints results, and how to read AI hardware claims. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/mlperf-training-v6-1-endpoints-explained
BibTeX
@online{kelahmet2026mlperf,
author = {Kelahmet, {\"O}zer Cem},
title = {{MLPerf Training v6.1 and the first Endpoints results, and how to read AI hardware claims}},
year = {2026},
month = {nov},
url = {https://www.ozercemkelahmet.com/articles/mlperf-training-v6-1-endpoints-explained}
}How this article connects
12 pages link to or from it. Drag to turn, click a dot to open it.
Version historyfirst version
- Published
Factual corrections are listed here with what changed. Wording and formatting edits are not listed.