TL;DR
IMO-CoT is introduced, a novel, selective, information-rich benchmark derived from International Mathematics Olympiad (IMO) problems, designed to evaluate CoT reasoning capabilities in LLMs.
Large language models (LLMs) have recently improved their problem-solving abilities and can solve complex mathematical problems with an increasing accuracy, necessitating the development of more challenging benchmarks. Over the years, the performance of LLMs on several benchmark datasets has also improved, motivating the development of evaluation frameworks that emphasize deeper algorithmic reasoning. This research presents Bench of Euler, a benchmark comprising 954 challenging mathematical and computational puzzles from the “Project Euler” collection, designed to evaluate complex multi-step mathematical reasoning. Solving these puzzles requires in-depth algorithmic deduction, as solutions based on brute force enumeration are mathematically infeasible. The benchmark is further classified into bands with respect to the problem difficulty to enable structured evaluation. Several existing open-source and proprietary, as well as base and reasoning, models have been evaluated on Bench of Euler, among which the highest-performing gpt-4o model by OpenAI achieved the highest accuracy of 43.21%. To evaluate the algorithmic efficiency, the models were prompted to generate Python code, which was subsequently executed in an isolated sandbox with strict timeout constraints. A failure analysis of the top-performing models revealed their inability to effectively optimize solutions, resulting in time-limit exceeded (TLE) errors during execution and outputs that often remained incomplete or approximate. Additionally, second-pass evaluations yielded an average performance improvement of approximately 2.09%, indicating limited gains from a second stochastic sample under the adopted decoding protocol.
This work presents a theoretical framework that reveals how reasoning steps can amplify error through three failure modes: incorrect sub-task decomposition, incorrect sub-task solving, and incorrect final answer summarization, and introduces structured interventions that adapt CoT generation according to the identified failure types.
Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs'ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.
Reason Popper-ly, a neurosymbolic framework that uses inductive logic programming (ILP) to learn relation composition rules from reasoning traces and deploys them as an online verifier for step-level correction, consistently improves terminal accuracy over standard CoT.
A novel RAIE taxonomy along four scaling dimensions is proposed, which optimizes the entire thought process through search algorithms and self-verification, and introduces a task-oriented guideline for choosing the best TTS strategy.
This work proposes ChainPrune, a novel reasoning path semantic structural optimization method to efficiently and controllably synthesize self-generated high-quality training data and incorporates a DPO-based preference learning method combined with supervised loss, effectively mitigating false reward suppression.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
