Skip to content

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

Sep 2026 · arXiv.org · Vol abs/2609.28449 · 1 citation · 48 references
Computer Science

TL;DR

SWE-Flux is introduced, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs.

Abstract

Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs. The benchmark covers singletest and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs shows that this task remains challenging. The best model achieves only 37% accuracy. Models perform better on localized behavior such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation. Finally, we show that the oracle-harvesting pipeline can generate fresh benchmark variants using input perturbation. It successfully harvests valid variants for almost 90% of the selected instances, and the resulting variants are substantially more challenging for the evaluated models.

View source

Similar papers

Open access Aug 2026

How well do LLMs understand code?

SemBench is introduced, a novel benchmark consisting of 1000 diverse C programs sourced from the CodeParrot GitHub-code dataset, with 15,404 semantic questions spanning six basic but fundamental properties: dead code-statement, data dependency, function reachability, dominator, dead code-loop, and liveness.

Jade Xu, Ren-Liang Sun, Zijian Ding et al. · 0 citations
#artificial intelligence Preprint Sep 2026

PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces

PetriBench is introduced, a compact, fully self-contained, and scalable benchmark for evaluating LLM reasoning over dynamic state spaces using Petri nets, a mature formalism for modeling real-world concurrent and distributed systems.

Pyrros Koussios, Benjamin Jäger, John Hua Yao et al. · 0 citations
#small language model Open access Oct 2026

T-REX: Teaching Large Language Models to Reason with Verbalized Execution Semantics

Large language models (LLMs) have shown strong performance in static code tasks like code search, summarization, and generation, but remain limited in dynamic code reasoning, which involves inferring how programs behave during execution without actually running them. This limitation stems from LLMs being trained on sta...

Yan Wang, Ling Ding, Jie-Chen Sun et al. · 0 citations
Open access Oct 2026

CAST: A Compiler-Based Framework for Systematically Testing LLM Compositional Safety

This paper proposes CAST, a systematic testing framework designed to evaluate the compositional safety of LLMs in the domain of malicious code, and demonstrates that CAST systematically exposes severe safety violations in strongly aligned models that resist conventional red-teaming.

Lu Yan, Zhuo Zhang, Xiang-Zhe Xu et al. · 0 citations
#artificial intelligence Book Open access Sep 2026

Predicting Program Exit Code with LLMs and Programming Language Semantics

This work evaluates open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits, showing that LLMs lean on pre-training priors rather than systematically applying the given rules.

Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.