Emergent Reasoning in Large Language Models: A Systematic Evaluation Across Task Complexity
TL;DR
It is argued that emergent reasoning in LLMs is a product of three factors: model size, prompting approach, and evaluation metric, and proposed implications for designing benchmarks and assessing capabilities are proposed.
Abstract
Large language models (LLMs) are said to exhibit “emergent” reasoning capabilities — ones that are virtually nonexistent in smaller models but suddenly emerge as soon as the model size surpasses a critical point. This claim has been at the heart of discussions on the capability forecasting, safety planning and evaluation methodology of LLM, but is disputed by recent research that suggests that the apparent emergence is merely an artifact of discontinuous evaluation metrics rather than a property of the underlying model. This paper offers a systematic comparison of reasoning behaviour for four model-scale classes (around 0.5B, 6B, 30B, and 70B+ parameters), and a taxonomy of five levels of task complexity ranging from factual recall to multiple-step arithmetic and logical reasoning to compositional generalization to open-ended planning. We use a benchmark set of 2,600 items sampled from existing reasoning corpora to evaluate accuracy for direct prompting, chain-of-thought (CoT) prompting, and self-consistency decoding and examine the evolution of accuracy curves as we increase model size and explore the three prompting methods. We find that accuracy decreases smoothly with increase in complexity and for each scale class, the rate of increase of accuracy is complexity-dependent: for low complexity tasks, accuracy is improved more gradually and predictably, whereas for multi-step or compositional tasks, accuracy shows sharp, threshold-like gains between the 6–8B and 30–70B classes, which are significantly amplified by CoT elicitation. We also demonstrate that much of this apparent sharpness can be eliminated—though not entirely—by replacing accuracy with a continuous partial credit measure, supporting both the emergence and measurement artifact explanations. Finally, we argue that emergent reasoning in LLMs is a product of three factors: model size, prompting approach, and evaluation metric, and propose implications for designing benchmarks and assessing capabilities. Keywords—large language models; emergent abilities; chain-of-thought reasoning; task complexity; benchmark evaluation; compositional generalization.