Inference-Time Scaling

Three budgets

Every inference-time method spends compute on one or more of:

• Depth: a longer single attempt, more steps of chain of thought, more moves in a search.
• Breadth: more independent attempts, as in self-consistency and best-of-N.
• Verification: checking, scoring or revising candidates, as with verifiers.

They fail differently. Breadth cannot recover a solution the generator never produces. Verification cannot select a correct answer absent from the batch. Depth cannot help if the chain heads the wrong way and nothing checks it. The search lab shows the first two directly: width 3 rescues the route to 19 at six moves, and no width finds it in five.

The evidence

OpenAI's o1 announcement reported that the model's performance improved consistently with more reinforcement-learning training compute and with more time spent thinking at test time. Established It gave the trend as plots without the underlying training details, so treat it as a reported observation for one model family.

Snell and colleagues compared searching against process-based verifiers with letting the model revise its answers, and found the most effective method depended on problem difficulty; choosing per prompt was more than 4× more efficient than a best-of-N baseline, and in a FLOPs-matched comparison test-time compute could outperform a 14× larger model on problems where the smaller model already had non-trivial success. Established

The condition in that last sentence is the point: extra inference compute helps most where the model is close to solving the problem already.

What a request actually costs

For a reasoning request, count:

• prompt tokens (prefill, once per sample unless cached);
• generated tokens, including hidden reasoning tokens, which are usually billed;
• number of samples;
• verifier or tool calls;
• latency, which grows with the longest sequential chain.

Tiny example. One attempt of 2,000 reasoning tokens versus 8 parallel attempts of 500 tokens each. The parallel option generates twice the tokens (4,000) but each sequence is a quarter as long, so wall-clock latency can be lower if the hardware runs the samples together. The KV cache is the other cost: eight concurrent sequences hold eight caches.

Allocate by difficulty

An easy request answered in 20 tokens and a hard one given 20,000 should not share a budget. A budget policy decides:

1. how to estimate difficulty (a classifier, the model's own uncertainty, disagreement among a few cheap samples);
2. which axis to spend on first;
3. when to stop, and what to report when the budget runs out without a verified answer.

Evaluate a system's budget policy, not just its model: which requests get more work, when it stops, and how it behaves at the limit. Interpretation

Reading results

When a paper or product reports a gain from more inference compute, ask: compute measured how (tokens, FLOPs, dollars)? Selection by what (vote, learned scorer, oracle)? Compared at matched compute against what baseline (more samples from a bigger model, a cheaper model with more samples)? And on which difficulty range?

Mini experiment

In the consensus lab, at 20% fresh accuracy, compare the verifier's success with 4, 16 and 32 candidates over a few seeds. Then estimate the cost multiplier of each budget relative to 4. Where would you stop if each extra candidate cost one cent and a wrong answer cost ten?