Bench of Euler: A Benchmark for Evaluating the Problem-Solving Abilities of Large Language Models

Abstract

Large language models (LLMs) have recently improved their problem-solving abilities and can solve complex mathematical problems with an increasing accuracy, necessitating the development of more challenging benchmarks. Over the years, the performance of LLMs on several benchmark datasets has also improved, motivating the development of evaluation frameworks that emphasize deeper algorithmic reasoning. This research presents Bench of Euler, a benchmark comprising 954 challenging mathematical and computational puzzles from the “Project Euler” collection, designed to evaluate complex multi-step mathematical reasoning. Solving these puzzles requires in-depth algorithmic deduction, as solutions based on brute force enumeration are mathematically infeasible. The benchmark is further classified into bands with respect to the problem difficulty to enable structured evaluation. Several existing open-source and proprietary, as well as base and reasoning, models have been evaluated on Bench of Euler, among which the highest-performing gpt-4o model by OpenAI achieved the highest accuracy of 43.21%. To evaluate the algorithmic efficiency, the models were prompted to generate Python code, which was subsequently executed in an isolated sandbox with strict timeout constraints. A failure analysis of the top-performing models revealed their inability to effectively optimize solutions, resulting in time-limit exceeded (TLE) errors during execution and outputs that often remained incomplete or approximate. Additionally, second-pass evaluations yielded an average performance improvement of approximately 2.09%, indicating limited gains from a second stochastic sample under the adopted decoding protocol.