Best-of-N: Inference-Time Compute for Small Language Models

Abstract

Technical note TR-2026-02, superseding TR-2026-01. We measure what happens when a frozen language model is sampled 128 times perproblem and the result is selected rather than taken as given. On GSM8K,Qwen2.5-0.5B-Instruct goes from 45.3% at a single sample to 66.5% undermajority voting, with no weights modified. Coverage - whether any trajectoryreaches the correct answer - reaches 93.5%. We separate the gain into the part attributable to selection (+20.2 points) andthe part attributable to simply having a trajectory that finished (+1.0points), and show that the second term is a property of the experimental setuprather than of the method: at a tighter token budget it was four times larger,and would conventionally have been reported as method gain. Selection is invariant under permutation of the trajectory pool, for everyselector other than random. This is not automatic: it requires an equivalencepartition built as a transitive closure, exactly-rounded summation of theweights, and tie-breaking that does not encode arrival order. The property isverified directly and is visible in the published output. The same protocol is applied to eight models, from SmolLM2-1.7B toQwen3.8-27B. Every one improves, by between +2.4 and +37.3 points, and thegain shrinks as the base model gets better. Coverage stays above what the votereturns on all eight, by a median of 9 points. All 25,600 trajectories are published in full, and every figure is recomputedfrom them by a script that re-extracts answers from the raw reasoning textrather than re-counting stored ones. This version supersedes TR-2026-01. The measurements in that report wereproduced by version 1.0.0 of the software, which contained defects in answerextraction and in the handling of truncated trajectories, and they are notreproducible with the corrected implementation. They should not be cited.Specifically withdrawn is the claim of a trained outcome reward model reachingROC-AUC 0.910 and a 16.6-point improvement over majority voting. Software: pip install bestofn - Apache License 2.0Code and data: https://github.com/voidlinestudios12-jpg/Interlace-AI