Skip to content
Open access

HeroBench: An Executable Benchmark for Long-Horizon Planning and Feasibility Reasoning in Large Language Models

2026 · IEEE Access · Vol 14, pp. 166725-166738 · 0 citations · 32 references

Abstract

Large language models (LLMs) perform well on many stepwise reasoning tasks, yet their ability to construct reliable plans over long horizons remains difficult to measure. Existing evaluations often use compact symbolic domains or allow execution-time feedback, making it harder to isolate failures in end-to-end plan construction. We introduce HeroBench, an executable benchmark for long-horizon planning and feasibility reasoning in a fully specified virtual world. Each task requires a model to jointly solve hierarchical resource dependencies, spatial routing, discrete equipment selection, numerical combat constraints, and, in extended settings, skill progression. In the primary zero-shot, single-generation protocol, the model must emit the complete action program before receiving any execution feedback. A deterministic simulator then measures goal completion, milestone progress, oracle-relative action efficiency, and attributable plan-level and action-level errors. Across 24 model configurations, reasoning-enabled systems generally achieve higher success, but performance declines as dependency depth and execution horizon grow. Exploratory infeasibility, repeated-sampling, progression, distractor, and semantic-obfuscation experiments further probe reachability judgment and robustness to representation changes. HeroBench provides a scalable testbed for studying whether improvements in local reasoning translate into reliable, globally consistent planning.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.