Jun 2026· Proceedings of the ACM on Software Engineering· Vol 3, pp. 2790 - 2812· 2 citations· 60 references
Computer Science
TL;DR
RepoReasoner is introduced, a benchmark for evaluating repository-level code reasoning that assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context.
Abstract
Recent large language models (LLMs) have shown strong performance on software engineering tasks, yet most existing benchmarks evaluate code reasoning at the function level, where all relevant information is localized. This setting fails to reflect real-world development, which requires reasoning across multiple files and complex dependency structures. We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning. It assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context. Our benchmark is constructed through a multi-stage pipeline that leverages dynamic tracing of pytest executions to obtain ground-truth call chains, along with LLM-based I/O rewriting to reduce memorization effects. We evaluate seven state-of-the-art LLMs. Even under oracle context, the best-performing model achieves only 69.1% Pass@1 on Output Prediction, indicating that cross-file reasoning remains a major challenge. In Call Chain Prediction, models exhibit high precision but low recall, suggesting limited multi-hop dependency understanding. Furthermore, performance drops on rewritten data reveal partial reliance on memorization, and longer contexts do not consistently improve results due to noise. These findings highlight fundamental limitations in current LLMs’ repository-level reasoning and motivate future work on structured architectural understanding and cross-file inference.
Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision remains underexplored because there is no standard notion of a step. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize. We propose STEP-KTODER, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests. Our method provides a code-specific instantiation of stepwise KTO, combining function-level process supervision with outcome-level feedback on the full program. We evaluate on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing that STEP-KTODER improves over outcome-only KTO and DPO. Further analysis shows that execution-based labels are essential: LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization. Code is available at: https://github.com/inechnech/STEP-KTODER.
Idris Nechnech, Sehwan Kim, Jimin Seo et al.· 0 citations
XREPOTEST is introduced, a multilingual repository-level benchmark for unit test generation spanning five underexplored languages: Rust, Go, Julia, PHP, and Ruby, and Invocation Rate is proposed to assess whether generated tests meaningfully exercise the intended functionality.
L. Dung, Dong Cao Van, Nam Le Hai et al.· 0 citations
Background: Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. Method: We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across languages.
Hui Sun, Anderson G. Uchôa, Rohit Gheyi et al.· 0 citations
This paper introduces CodeVulReason, a unified framework for enhancing the reasoning capabilities of large language models (LLMs) in code vulnerability detection (CVD). To address the scarcity of high-quality chain-of-thought (CoT) supervision, we automatically construct task-specific reasoning data using frontier reasoning models to generate and curate reasoning traces from benchmark datasets, including DiverseVul and MixedVul. We further propose S-LoRA, a parameter-efficient fine-tuning method that optimizes Low-Rank Adaptation (LoRA) rank allocation through a statistically grounded stability-aware importance scoring mechanism. In addition, we integrate S-LoRA with group relative policy optimization (GRPO) to further refine task-specific reasoning behavior. Experimental results show that CodeVulReason achieves state-of-the-art performance on two benchmark datasets, including an accuracy of 0.785 on DiverseVul, while also improving the interpretability of model predictions through explicit reasoning traces.
Zhengye Li, Kenny Zhu· Journal of King Saud Univers...· 0 citations
Repository-level code completion requires understanding complex cross-file dependencies, yet existing approaches struggle to balance context richness with the sub-200ms latency requirements of real-time IDE integration. We present RepoFuse, a dual-context framework deployed at Ant Group1 serving 10,000+ developers across 50+ million lines of code. In a production deployment at Ant Group, RepoFuse maintained an average latency of 156ms and improved completion acceptance rate by 18%. This was accomplished through systematic categorization of cross-file context into semantic context (structural dependencies via lightweight static analysis) and similar context (functionally analogous patterns via similarity search), combined with a relevance-guided selection algorithm that optimizes the efficiency-accuracy trade-off. Benchmark evaluation on CrossCodeEval demonstrates 4.26 and 3.01 improvements in exact match scores for Python and Java respectively, while using only 25% of baseline token requirements. More significantly, our one-week A/B test with 80,000 completion requests revealed that developers particularly valued the dual-context approach when working across microservice boundaries, with 73% reporting reduced context switching between files. The system successfully operates under production constraints with distributed caching, real-time graph updates, and p95 latency under 190ms, demonstrating that principled context categorization enables practical repository-level completion at industrial scale. We have open-sourced RepoFuse at GitHub2.
Ming Liang, Xiaoheng Xie, Gehao Zhang et al.· SIGSOFT FSE Companion· 0 citations
Large language models are known to be sensitive to prompt formulation. Even minor variations in wording can substantially degrade performance. This sensitivity reveals an opportunity: if prompt phrasing can harm performance, can it be used to improve it? To investigate this question, we introduce a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance. We then propose DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures. A key strength of our approach lies in its generality: the evolved rules are error-agnostic, reusable across problems, and transferable across models. We evaluate DUALFIX against execution-feedback repair baselines across three models on two challenging benchmarks, LiveCodeBench and APPS. Our results show that the evolved transformations fix from 10-30% of failing cases, including 12-17% of failures that execution-based repair alone cannot resolve. Overall, DualFix recovers up to 30% of baseline failures and fixes 3-5 times more failing cases than Self-Fix across all evaluated settings. Furthermore, we also show that rules evolved on one model transfer zero-shot to other models, outperforming execution-feedback repair without any re-optimization.
Amal Akli, Melissa Akli, Cedric Richter et al.· 0 citations