Gemini 4 Carbon tops MathArena: how to read scores, costs and intervals on a math leaderboard
Google's Gemini 4 Carbon now sits first on MathArena, ahead of Claude Fable 5.2 and GPT-6.1 Sol. The ranking is real but narrow, and the leaderboard itself is a good lesson in how to measure reasoning progress honestly.
- Reading time
- 17 min
- Sections
- 8
- Code examples
- 1
- Tables
- 2
- Sources
- 12
- Published
- 17 Nov 2026
On 13 November 2026, MathArena added results for Google's Gemini 4 Carbon and Anthropic's Claude Fable 5.2. Carbon went straight to first place. Its expected performance is 85.4%, with an interval of plus or minus 3.0 points. Claude Fable 5.2 came second at 82.8%, and OpenAI's GPT-6.1 Sol third at 80.6%. The Decoder reported the result on 14 November and called Carbon "the first non-OpenAI model in a while" to top the benchmark.
The story underneath is more useful. The three leading models have overlapping intervals, so the order is not a clean separation. Their cost per problem ranges from about one dollar to about forty dollars. And Carbon's most common failure, according to the report, was not a wrong answer at all. It was running out of time.
MathArena publishes its methods in two arXiv papers and a series of blog posts, which makes it a good case study in reading any reasoning leaderboard. For the general version of this lesson, read a score is only as good as its report.
What happened
Anthropic launches Claude Fable 5.2 (9 November 2026)
Anthropic released Claude Fable 5.2 on 9 November, alongside a restricted model called Claude Mythos 5.2. Fable 5.2 is available on all platforms under the API name claude-fable-5-2. Anthropic lists it at $8 per million input tokens and $40 per million output tokens. MathArena tested it at the Max effort level.
Anthropic also says a team of about 1,000 Fable 5.2 agents completed a Lean formalization of Fermat's Last Theorem in 30 hours with minimal human help. Anthropic notes that this was a single run per model. Treat it as a company claim, not a benchmark result.
Google opens Gemini 4 Carbon to paying developers (10 November 2026)
Google first announced Gemini 4 Carbon on 28 October 2026 as "our next frontier model". At that point it went only to a subset of cyber defenders. Google said Carbon is built on Gemini 4 Argon, which launched two weeks earlier. On 10 November, Google made Carbon available through the Gemini API in Google AI Studio and Gemini Enterprise for all paid developers and businesses. Launch pricing is $2 per million input tokens and $10 per million output tokens. Google says the price rises to $4 and $20 after the launch period ends on 30 November 2026. MathArena lists 10 November as Carbon's release date and tested it at the "high" setting.
MathArena adds both models (13 November 2026)
MathArena's news feed for 13 November reads: "We added Claude Fable 5.2 and Gemini 4 Carbon results to MathArena!" The leaderboard now shows the top of the table below. The site lists 137 models in total, with 119 ranked.
| Rank | Model | Provider | Expected performance | Expected cost per problem |
|---|---|---|---|---|
| 1 | Gemini 4 Carbon (high) | 85.4% ±3.0% | $15.70 ±$1.33 | |
| 2 | Claude-Fable-5.2 (max) | Anthropic | 82.8% ±5.3% | $40.20 ±$2.59 |
| 3 | GPT-6.1 Sol (max) | OpenAI | 80.6% ±2.7% | $1.07 ±$0.018 |
| 15 (best open model) | Kimi K3.1 Pro (max) | Moonshot AI | 54.3% ±5.8% | $3.70 ±$0.27 |
The best open-weight model, Kimi K3.1 Pro, is about 31 points behind the leader. We covered that model's design in sizing giant open MoE models.
The Decoder reports cost and timeouts (14 November 2026)
The Decoder's Matthias Bastian wrote up the result on 14 November, citing MathArena's Jasper Dekoninck. The report says Carbon and Fable 5.2 cost roughly 15 and 40 times as much as GPT-6.1 Sol at the tested settings. It says Carbon's most common failure mode is exceeding the time limit. Carbon "almost never gives up on a task". In one case, it ran a script for more than eleven hours without checking on its progress.
How MathArena turns many runs into one number
Behind the single list sit many competitions, many models and thousands of attempts. MathArena uses three tools to combine them.
Four runs per problem
Ask a reasoning model the same question twice and you may get two different answers. MathArena handles this with repetition. Its site says: "Each model runs four times on each problem." The original MathArena paper explains that accuracy is the average over those runs. There is no majority voting. So a model that solves a problem in two of four runs gets half credit for it.
This measures what a user gets from one try. Majority voting would report a different, more expensive product.
Expected performance and item response theory
Not every model is run on every competition. New benchmarks appear and old models get retired, which leaves holes in the results table. A plain average would be unfair: a model tested only on easy sets would look strong.
MathArena fills the holes with item response theory, or IRT. The site says: "For missing results, we estimate the score with a two-parameter item-response theory model." The platform paper, published on arXiv in May 2026, gives the details. Each model gets one capability number. Each question gets a difficulty and a discrimination. The chance that a model solves a question rises as its capability rises above the question's difficulty. Discrimination controls how sharply that chance rises. A high-discrimination question separates strong and weak models cleanly.
Think of a high-jump competition where not every athlete tried every bar height. If you know how high each athlete cleared on the bars they did attempt, and how hard each bar is in general, you can estimate how each athlete would do on the bars they skipped. IRT does the same thing for models and math questions. Expected performance is then the average predicted success across all questions in the non-deprecated competitions.
Two cautions apply. First, imputed scores are estimates. The paper says the method is "robust to various modeling choices", but it does not report how well it predicts held-out results. Second, MathArena sets a floor: "A model needs to have at least 42 answered questions to be included in the expected performance plots." That stops a model with almost no data from appearing with a confident score.
Intervals, not just points
Every score on the leaderboard carries a plus-or-minus figure. The platform paper says these are 95% confidence intervals using the normal approximation. The formula treats each solution as a coin flip that lands correct with some probability. The more solutions you count, the narrower the interval.
Think of a poll. A poll that says 52% to 48% with a three-point margin of error has not called the election. The same logic applies here. Carbon's interval runs from about 82.4% to 88.4%. Fable 5.2's runs from about 77.5% to 88.1%. Sol's runs from about 77.9% to 83.3%. All three ranges overlap. The data are consistent with any of the three being best on the underlying task.
Overlapping intervals are a rough guide, not a formal test. The original MathArena paper builds rank intervals with a paired permutation test, which compares models on the same questions and can detect smaller gaps. Still, the plain reading is fair: first place is a lead, not a clear win. Under the normal approximation, Fable 5.2's wider interval at a similar score points to fewer counted solutions than the others. For more on picking and reading metrics, see evaluation metrics.
Why the cost column matters as much as the score
The leaderboard shows expected cost next to expected performance. MathArena defines it as "the weighted average per-problem cost over non-deprecated, non-Euler competitions". When a cost is missing, the platform paper says it fits a simple model to the observed costs and predicts the gap.
Cost per problem depends on the price of each token and how many tokens the model spends thinking. It is like a taxi fare: a cheap rate on a long detour can cost more than a higher rate on a direct road.
| Model | Score | Cost per problem | Rough cost ratio vs Sol |
|---|---|---|---|
| Gemini 4 Carbon (high) | 85.4% | $15.70 | about 15x |
| Claude-Fable-5.2 (max) | 82.8% | $40.20 | about 38x |
| GPT-6.1 Sol (max) | 80.6% | $1.07 | 1x |
| Kimi K3.1 Pro (max) | 54.3% | $3.70 | about 3.5x |
Kimi K3.1 Pro costs more per problem than GPT-6.1 Sol while scoring far lower. Also notice the settings. Carbon ran at "high", Fable 5.2 and Sol at "max". These are vendor effort settings, and they change both score and cost. A leaderboard row describes one model at one setting, not the model in general. For background on why effort settings change results, see test-time compute.
For teams that pay the bill, the question is which model clears their accuracy bar at the lowest cost per solved task. On these figures, the 4.8-point gap between Carbon and Sol costs about $14.60 extra per problem, and that gap sits inside overlapping intervals. Whether that trade is worth it depends on the job. For setting spending limits, see cost and latency budgets and per-employee AI budgets.
Keeping the test fresh: contamination, proofs and Lean
A benchmark only measures reasoning if the model has not already seen the answers. Once a problem and its solution are online, they can end up in training data.
Fresh competitions
MathArena's founding idea, set out in the 2025 paper "MathArena: Evaluating LLMs on Uncontaminated Math Competitions", is to test models on competition problems right after they are released. The authors found "strong signs of contamination in AIME 2024". They compared model scores on the 2024 and 2025 contests against human performance quantiles. On AIME 2024, most models sat 10% to 20% above the level that comparison would predict. One model, QwQ-Preview-32B, beat its expected level by nearly 60%. The gap was much smaller on HMMT.
Moving past final answers
Final-answer problems are easy to grade. You check one number. But they are now too easy for frontier models. In a blog post on 12 May 2026, the MathArena team said they tried to build a new hard set from 176 recent final-answer problems. Gemini 3.1 Pro solved 162 of them in all four attempts. The team concluded that "final-answer competition problems are no longer an effective source of hard benchmark tasks for frontier models." The site now marks several older competitions as deprecated, and the expected-performance score averages only the non-deprecated ones.
The replacements test harder skills:
- Proof competitions. Models write full proofs, which judges grade. In the original paper, two independent judges grade each anonymized solution against a scheme with partial credit. The platform paper reports that GPT-5.5 scored 98% on the 2026 USA Math Olympiad.
- Research problems. ArXivMath draws questions from recent arXiv papers. BrokenArXiv checks whether a model will still claim a proof when the statement is subtly false.
- Formal proofs in Lean. Lean is a proof assistant. A computer checks every step, so no human judge is needed and no bluffing survives. ArXivLean, launched on 21 April 2026, took 41 theorems from papers posted in March 2026. At launch, "even the strongest models formally prove only 17.1% of research theorems." The team plans to refresh it every three months to limit contamination.
The trade-off is clear. Final answers are cheap to check but leak and saturate. Graded proofs are rich but slow. Lean proofs are strict and automatic, but formalizing a statement is itself hard work. A sound benchmark needs a mix. If you build tests for your own tasks, our evals guide is a starting point.
Timeouts are part of the measurement
A model that never gives up sounds like a virtue. In a timed test, it can be a defect.
MathArena has dealt with time and failure before. On LeanLean October, published on 6 November 2026, each task had a 12-hour time limit and a $100 budget. Thirty-two of 160 runs ended at a limit. Gemini 4 Argon, the model Carbon is built on according to Google, accounted for 22 of them: 10 at the time limit and 12 at the budget. GPT-6.1 Sol hit the time limit three times and the budget twice, but all five of those submissions still passed verification. This does not prove Carbon and Argon fail for the same reason. It does show that time management now separates frontier agents.
How a benchmark treats failed runs changes the score. In a January 2026 post, MathArena showed that retrying failed requests until they succeed can overestimate performance by up to 5%. The reason is subtle. Failed requests tend to be the long ones, and long outputs are less often correct. Retrying acts like a filter that keeps the short, confident answers. The team now reports the average number of retries and recommends that every evaluator state its retry policy.
So read Carbon's timeouts two ways. A run that does not finish cannot score. And in production, an agent that runs a script for eleven hours without checking progress burns compute and blocks other work. Good agent harness design sets wall-clock limits, progress checks and kill switches, whatever the model.
What it means in practice
The leaderboard gives three practical lessons.
First, the frontier is crowded. Three labs sit within five points, with overlapping intervals. A claim that one model "is best at math" goes beyond the data.
Second, the top three differ by under five points in score but by roughly a factor of 38 in cost per problem. For most buyers, cost per solved task will matter more than rank.
Third, behaviour under limits matters. A model that runs past its time budget fails timed tasks, and a single accuracy number does not show that. You only see it in the failure analysis.
The same lesson runs through our look at METR's AI evaluations: good measurement reports uncertainty, cost and failure modes alongside the headline.
What to do now
- Read the interval before the rank. If intervals overlap heavily, treat the models as tied and decide on other factors.
- Compare cost per solved task, not price per token. Divide cost per problem by accuracy to get a rough cost per correct answer, and compare that.
- Note the effort setting. Write down "high" or "max" next to every score you quote. A different setting is a different product.
- Test at your own effort level. A result at max effort does not describe a system you run at medium.
- Set wall-clock and budget limits on agents. Give every agent run a time cap, a spending cap and a progress check, as LeanLean does with its 12-hour and $100 limits.
- Log timeouts and retries separately. Count them as their own outcome, not as silent retries, so your internal numbers do not drift upward the way MathArena's retry study shows.
- Use fresh or private test items. Write some test problems after the model's release, or keep them private, to limit contamination.
- Prefer checkable outputs where you can. For maths, code and data work, use automatic verifiers, such as Lean, unit tests or schema checks, rather than trusting a model's own claim of success.
Risks and open questions
Imputed scores can hide weak spots. IRT fills gaps with predictions, and one capability number may smooth over a narrow weakness. Check which competitions a model actually ran.
The aggregate mixes very different tasks. Expected performance averages across all non-deprecated competitions, which test different skills. A single percentage cannot tell you whether a model is better at checking proofs or at finding numerical answers.
Vendor settings and API stability shift results. The same model can score differently at another effort setting or under another retry policy.
Company claims are not benchmark results. Anthropic's Fermat formalization is a single-run company claim, not an independent, repeated measurement.
Watch next: whether MathArena publishes a detailed breakdown of Carbon's timed-out runs, whether Google adjusts Carbon's behaviour on long tasks, how the top three's intervals change as more competitions are added, and whether open-weight models close the roughly 31-point gap to the leader.
Frequently asked questions
Is Gemini 4 Carbon clearly the best math model?
Not clearly. It has the highest expected performance on MathArena at 85.4%, but its interval overlaps with those of Claude Fable 5.2 and GPT-6.1 Sol. The ranking is a lead, not a decisive separation.
Why is Carbon so much more expensive per problem than GPT-6.1 Sol?
Cost per problem depends on both the price per token and how many tokens a model spends. MathArena reports Carbon at $15.70 per problem and Sol at $1.07, roughly 15 times apart.
What does item response theory do on MathArena?
It estimates scores for questions a model was never run on. Each model gets a capability value and each question a difficulty and a discrimination value. The fitted model predicts the chance of success on every question, and the average of those predictions is the expected performance.
Why run each model four times on each problem?
Reasoning models give different answers on different runs. Averaging four runs gives a steadier estimate of what a user gets from a single attempt. MathArena does not use majority voting, so the score reflects one-shot reliability.
What is the difference between final-answer and Lean tracks?
Final-answer tracks check only the end result, such as a number, which is cheap but easy to saturate and leak. Lean tracks require a full formal proof that a computer checks step by step. MathArena now says final-answer competitions are no longer an effective source of hard tasks for frontier models, and it points to proofs, research problems and Lean instead.
Summary
Gemini 4 Carbon took first place on MathArena on 13 November 2026 with an expected performance of 85.4%, ahead of Claude Fable 5.2 at 82.8% and GPT-6.1 Sol at 80.6%. The three intervals overlap, so the order is not a clean win. Costs differ far more than scores: about $1 per problem for Sol, about $16 for Carbon and about $40 for Fable 5.2. MathArena's methods show careful measurement: four runs per problem, item response theory to fill gaps, published intervals, fresh problems and a shift to proofs and Lean. Carbon's reported timeouts show that failure modes belong in the report. Compare cost per solved task, note the effort setting, and cap long-running agents. For background on these models, see our guide to reasoning models.
Sources
- MathArena: Leaderboard and news
- MathArena: Models
- The Decoder: Gemini 4 Carbon takes the MathArena crown but costs 15 times more than GPT-6.1 Sol
- arXiv: MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- arXiv: Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
- Google: Gemini 4 Carbon: our next frontier model
- Google: Gemini 4 Carbon ya está disponible
- Anthropic: Claude Fable and Mythos 5.2
- MathArena: Farewell to Final-Answer Competition Problems as Frontier Benchmarks
- MathArena: ArXivLean: How Well Can LLMs Formally Prove Research Math?
- MathArena: LeanLean October: Proof Compression at Even Larger Scale
- MathArena: The Hidden Effect of Retrying Requests in LLM Evaluation
Continue learning
Cite this article
Kelahmet, Ö. C. (2026, November 17). Gemini 4 Carbon tops MathArena: how to read scores, costs and intervals on a math leaderboard. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/gemini-4-carbon-matharena-leaderboard
BibTeX
@online{kelahmet2026gemini,
author = {Kelahmet, {\"O}zer Cem},
title = {{Gemini 4 Carbon tops MathArena: how to read scores, costs and intervals on a math leaderboard}},
year = {2026},
month = {nov},
url = {https://www.ozercemkelahmet.com/articles/gemini-4-carbon-matharena-leaderboard}
}How this article connects
10 pages link to or from it. Drag to turn, click a dot to open it.
Version historyfirst version
- Published
Factual corrections are listed here with what changed. Wording and formatting edits are not listed.