Test-time compute

Overview

For most of the history of large language models, the way to get a better model was to spend more compute before deployment: more parameters, more training data, longer training. Once the model was trained, answering a question cost roughly the same each time. Test-time compute changes that picture. It is the idea that you can also improve answers by spending more computation at the moment of answering, on the specific question in front of you.

There are several ways to spend that extra effort. The model can write out intermediate reasoning before it commits to an answer. You can ask it the same question many times and take a vote. You can generate many candidate answers and let a second model, a verifier, pick the best one. Or you can run a structured search over partial solutions, keeping the promising branches and pruning the rest. All of these turn extra tokens, and therefore extra money and time, into better accuracy on hard problems.

An everyday analogy: a student taking an exam can write the first answer that comes to mind, or they can work through the problem on scrap paper, try two different methods, check that both give the same number, and only then write it down. The student's knowledge has not changed between the two approaches. What changed is how much effort they spent on that one question. Test-time compute is the scrap paper, the second method and the checking, applied to a model.

What it is

Two broad families appear in this table. Parallel methods generate many answers side by side and then select or aggregate them; self-consistency and best-of-n are the classic examples. Sequential methods make a single answer longer and more careful, by thinking step by step, revising, or extending the reasoning when the model wants to stop. Search methods sit in between: they grow several partial solutions in parallel but extend each one sequentially.

A second distinction is whether the extra compute is learned or bolted on. A reasoning model has been trained, usually with reinforcement learning, to produce long internal reasoning by itself, so the extra compute is part of the model's behaviour. Voting, best-of-n and search are wrappers you can put around almost any model without retraining it. In practice the two are combined: a reasoning model can still be sampled several times and voted on.

Where it came from

The roots are in prompting research. In 2022, Jason Wei and colleagues at Google showed that giving a model worked examples that spell out intermediate reasoning, which they called chain-of-thought prompting, sharply improved results on arithmetic and commonsense problems. The same year Takeshi Kojima and colleagues found that simply adding "Let's think step by step" to a prompt had a similar effect without examples. Writing more tokens before the answer was, in effect, spending more compute on the question.

Xuezhi Wang and colleagues then introduced self-consistency (arXiv 2022, ICLR 2023): sample several chain-of-thought answers and pick the most common final answer. On the verifier side, Karl Cobbe and colleagues at OpenAI released the GSM8K maths dataset in 2021 in "Training Verifiers to Solve Math Word Problems", training a model to judge candidate solutions and using it to pick among many samples. In May 2023, Hunter Lightman and colleagues at OpenAI published "Let's Verify Step by Step", reporting that process supervision, which gives feedback on each reasoning step, significantly outperformed outcome supervision on the MATH dataset, and releasing PRM800K, a set of about 800,000 step-level human labels.

Search over reasoning steps became a research theme with Tree of Thoughts (Shunyu Yao and colleagues, 2023), which lets a model explore and evaluate branching lines of thought. In July 2024, Bradley Brown and colleagues published "Large Language Monkeys", showing that coverage, the share of problems solved by at least one sample, keeps rising across roughly four orders of magnitude of sample counts, often following a log-linear trend. In August 2024, Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar argued that a compute-optimal strategy, which allocates test-time compute according to how hard each prompt is, can be more than four times as efficient as a plain best-of-n baseline.

The idea reached products on 12 September 2024, when OpenAI introduced o1 and reported that its performance improved both with more reinforcement learning during training and with more time spent thinking at inference. DeepSeek-R1, released in January 2025 and later published in Nature, showed that long reasoning behaviour, including self-reflection and verification, could be encouraged through reinforcement learning with open weights. In January 2025, Niklas Muennighoff and colleagues published "s1: Simple test-time scaling", which fine-tuned Qwen2.5-32B-Instruct on just 1,000 curated reasoning examples and introduced budget forcing, a simple way to control thinking length. These systems are covered on the reasoning models page.

How it works

Reasoning tokens

The simplest form of test-time compute is letting the model generate more tokens before it answers. A transformer does a fixed amount of work per token, so a longer reasoning trace means more total computation on the problem. The intermediate tokens also act as working memory: the model can write down a partial result and condition on it later, which a single forward pass cannot do. Reasoning models are trained to produce these traces automatically, often in a separate "thinking" section that may be hidden or summarised for the user.

Sampling and voting

Self-consistency samples several reasoning paths at a non-zero temperature, extracts the final answer from each, and returns the most frequent one. It works because a correct line of reasoning tends to arrive at the same answer from different directions, while mistakes scatter across many different wrong answers. It needs no extra model, but it only works when answers can be compared, such as numbers, multiple-choice letters or short labels.

Best-of-n with verifiers

Best-of-n generates n candidates and lets a scorer choose. The scorer can be an outcome reward model, which rates a finished answer; a process reward model, which rates each step; an LLM acting as a judge; or an exact checker such as a unit test suite or a proof assistant. When an exact checker exists, extra samples convert into accuracy very reliably. When the scorer is a learned model, it can be fooled, and selection quality tends to plateau.

Search

Search methods treat reasoning as a sequence of steps. At each step the system proposes several continuations, scores them with a step-level verifier, keeps the best few (beam search) or explores a tree with look-ahead, and continues. This spends compute where it matters, on promising partial solutions, instead of finishing many doomed attempts.

The key formulas

Self-consistency:   â = argmax_a  Σᵢ 1[ aᵢ = a ]            for i = 1 … n
Best-of-n:          ŷ = argmax_{y ∈ {y₁…yₙ}}  V(x, y)
Coverage (pass@n):  pass@n = 1 − (1 − p)ⁿ
Compute per query:  C ≈ n × (tokens per sample) × (cost per token)

In the first line, aᵢ is the final answer extracted from the i-th of n sampled reasoning paths, 1[·] counts a match, and â is the most common answer. In the second, x is the prompt, y₁ to yₙ are candidate answers, V is a verifier or reward model score, and ŷ is the chosen answer. In the third, p is the chance that one independent sample is correct, so pass@n is the chance that at least one of n samples is correct, which is the ceiling a perfect verifier could reach. In the last, C is the total cost, which grows linearly with the number of samples and the length of each one.

Budget control

Because thinking costs money and time, systems need a way to set how much of it to do. Commercial APIs expose settings such as a reasoning effort level or a maximum number of thinking tokens. The s1 paper's budget forcing works at decode time: to stop early, the system forces the end of the thinking section; to think longer, it suppresses the end and appends the word "Wait", which often prompts the model to re-check its work. Adaptive strategies go further and estimate difficulty first, giving easy prompts a small budget and hard ones a large budget.

Key concepts and terms

A worked example

Imagine an illustrative internal tool that answers maths questions for an engineering team. A single greedy answer from the model is correct about 60 per cent of the time on the team's test set. Each answer uses about 800 output tokens.

First, the team tries self-consistency with five samples. The model now writes about 4,000 output tokens per question, five times the cost, and the five answers are compared. On a question where three samples say 42, one says 40 and one says 48, the system returns 42. On the test set, accuracy rises, but questions where the model is consistently wrong stay wrong: voting cannot fix a systematic mistake.

Next, they check coverage. With 16 samples, at least one sample is correct on far more questions than the majority vote gets right. That gap tells them a good selector would help. Because many questions have an answer that can be checked numerically by substituting it back into the problem, they write a small checker and use best-of-16 with the checker as the verifier. Where the checker applies, accuracy jumps close to the coverage ceiling. Where it does not, they fall back to voting.

Finally, they add a budget rule. A cheap first pass answers each question once. If two quick samples agree and the checker passes, the system stops. Only questions with disagreement get the full 16 samples. Average cost drops well below the flat best-of-16 cost, while accuracy stays close to it. The lesson is the one Snell and colleagues formalised: spend compute where the difficulty is.

Strengths and limitations

Strengths:

• Improves accuracy on hard reasoning tasks without retraining or a bigger model.
• Lets one deployed model serve both cheap, fast requests and expensive, careful ones.
• Works especially well where answers can be checked automatically, such as code with tests or maths with exact answers.
• Wrappers like voting and best-of-n are simple to build and easy to reason about.
• Intermediate reasoning can make some errors easier to spot and debug.

Limitations:

• Cost and latency grow roughly linearly with samples and thinking length.
• Gains flatten: voting and learned verifiers plateau, and a weak base model cannot be rescued by sampling alone.
• Learned verifiers can be gamed, so best-of-n with a large n may pick confident but wrong answers.
• Majority voting needs comparable final answers and does not suit open-ended writing.
• Reasoning traces are not guaranteed to be a faithful account of how the model reached its answer.
• Long thinking on easy questions wastes money and can even introduce errors, a pattern often called overthinking.

How it compares with alternatives

The useful mental model is that training-time and test-time compute are two budgets you can trade between. A smaller model with generous thinking can beat a larger model answering instantly on problems it can partly solve, but no amount of thinking helps on problems far beyond the model's knowledge. Speculative decoding is listed because it is often confused with test-time compute; it makes generation faster without changing what is generated.

Practical guidance and best practices

1. Measure the baseline first: single-sample accuracy, cost and latency on a representative test set.
2. Measure coverage at several sample counts. A large gap between coverage and majority-vote accuracy means a better selector will pay off.
3. Use exact verifiers wherever you can: unit tests, schema validation, numeric checks or a solver. They beat learned reward models.
4. Start with the provider's reasoning effort or thinking budget settings before building your own sampling machinery.
5. Route by difficulty. Send easy requests through a cheap path and escalate only when signals such as disagreement or failed checks appear; see LLM routing.
6. Set hard limits on tokens and wall-clock time per request, and record them in your cost and latency budgets.
7. Evaluate the whole pipeline, not just the model, with an evals suite that tracks accuracy per unit of cost.
8. Watch for verifier exploitation by spot-checking the answers best-of-n selects at high n.
9. Reuse shared prompt prefixes across samples so the repeated part is not recomputed; see prompt caching.

Common mistakes

• Turning on maximum reasoning effort for every request, including trivial ones, and paying for thinking that adds nothing.
• Using majority voting on free-text answers where no two samples are ever literally identical.
• Trusting a learned reward model at large n without checking whether its top picks are actually correct.
• Reporting accuracy gains without the matching cost and latency, which hides whether the trade was worth it.
• Treating the visible reasoning trace as proof that the answer is right.
• Sampling at temperature zero and expecting diversity; identical samples make voting pointless.

Frequently asked questions

Is test-time compute the same as a reasoning model?

No. Test-time compute is the general idea of spending more computation per request. A reasoning model is one way to do it, where the model has been trained to generate long reasoning by itself. Voting, best-of-n and search are other ways, and they can be applied to reasoning and non-reasoning models alike.

Does more thinking always give better answers?

No. Accuracy usually rises with more compute and then flattens, and on easy questions extra thinking can waste money or introduce new mistakes. The benefit is largest on problems the model can partly solve, where it sometimes succeeds and sometimes fails.

When should I use best-of-n instead of majority voting?

Use best-of-n when you have a reliable way to score answers, ideally an exact checker such as unit tests. Use majority voting when answers are short and comparable but no checker exists. If you have a decent scorer and comparable answers, weighted voting combines both.

How do I control the cost?

Cap the thinking budget, use an effort setting, limit the number of samples, and escalate only uncertain cases. Track cost per correct answer rather than cost per request, so that you can see whether extra effort is paying for itself.

Can test-time compute replace a larger model?

Sometimes. Research by Snell and colleagues found that, in a compute-matched comparison, extra test-time compute could beat a much larger model on problems where the smaller model already had a reasonable success rate. On problems far beyond the smaller model's ability, the larger model still wins.

Is the reasoning trace a reliable explanation?

Not necessarily. The trace is generated text that helps the model reach an answer, but it is not guaranteed to reflect the computation that actually produced the answer. Some providers also hide raw traces and show summaries instead.

Glossary

• Test-time compute: computation spent per request at inference time.
• Chain-of-thought: written intermediate reasoning before a final answer.
• Self-consistency: majority vote over several sampled reasoning paths.
• Best-of-n: picking the highest-scoring of n candidate answers.
• Verifier: any model or program that judges whether an answer is good.
• Process reward model: a scorer that rates individual reasoning steps.
• Outcome reward model: a scorer that rates only the final answer.
• Coverage: the share of problems solved by at least one sample.
• Budget forcing: stopping or extending reasoning at decode time to control its length.
• Overthinking: spending long reasoning on a question that did not need it.

Summary

Test-time compute is the practice of spending extra computation on each request to get a better answer, instead of relying only on what was built in during training. It ranges from simple chain-of-thought and majority voting to best-of-n with verifiers, step-level search and trained reasoning models with adjustable thinking budgets. It works best on problems with checkable answers and on questions the model can partly solve, and its gains flatten as compute grows. Because cost and latency rise with every extra token and sample, the practical skill is allocation: measure coverage, use exact verifiers where possible, give hard questions more effort and easy ones less, and judge the whole system by accuracy per unit of cost.

Related

It belongs to Frontier topics in my stack, under Generative AI and LLMs. Others in the same group: