#artificial intelligence
Sep 2026
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
SWE-Flux is introduced, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs.
Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband et al.
· arXiv.org · 1 citation