A large-scale empirical study comparing representative code translation techniques across methodological paradigms and translation granularities, which shows that LLMs and LLM-based methods generally outperform learning-based methods in method-level correctness, although similarity metrics alone do not reliably reflect functional correctness.
Abstract
Automated code translation is increasingly important for software evolution, yet the relative strengths and limitations of learning-based and large language model (LLM)-based techniques remain insufficiently understood. To address this gap, we conduct a large-scale empirical study comparing representative code translation techniques across methodological paradigms and translation granularities. We evaluate learning-based methods, LLM-based methods, and general-purpose LLMs on multilingual method-level and class-level benchmarks involving multiple programming languages. Our analysis considers executable correctness, code similarity, translation direction, translation granularity, and failure patterns. The results show that LLMs and LLM-based methods generally outperform learning-based methods in method-level correctness, although similarity metrics alone do not reliably reflect functional correctness. Translation direction substantially affects performance, particularly when translating between languages with different type-system characteristics. Class-level translation remains considerably more difficult than method-level translation because it requires preserving global semantics, interfaces, member relationships, and cross-method dependencies. Our error analysis further shows that static semantic errors and logical errors are the primary challenges in existing code translation systems. These findings provide empirical evidence and practical guidance for developing more robust, type-aware, structure-aware, and context-aware code translation techniques.
This work investigates whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity, and identifies a difficulty-dependent breakdown in equivalence judgment.
Hui Sun, Anderson G. Uchôa, Rohit Gheyi et al.· 0 citations
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
Code translation, as a challenging and fundamental task, is increasingly relying on large language models (LLMs). However, LLMs often give seemingly plausible but fallacious translations, misleading and even deceptive to debugging developers. We propose tHinter, an automated approach that frames translation error localization as a differential testing task. By fuzzing and a novel Bayesian-inspired algorithm that integrates developer heuristics into SBFL, tHinter thoroughly exposes translation errors and effectively pinpoints functionality-twisting errors. We evaluate tHinter in a Python to C++ scenario, a representative and challenging task for migrating high-level logic to performance-critical systems. Experiments show that tHinter achieves the highest F1 in localizing translation errors, outperforming all baselines. When debugging LLM-translated code with human efforts, it averagely reduces 71% lines developers need to review. Furthermore, we provide a comprehensive evaluation reference beyond traditional fault localization metrics. We integrate mixed-factorial user studies, expert validation, and SWOT-based strategic analysis to assess the perceived helpfulness and resilience within the rapidly evolving LLM landscape, offering a methodologically rigorous example for researchers in the context of AI-augmented software engineering.
Shengnan Wu, Xin-Yu Sun, Xin Wang et al.· ACM Transactions on Software...· 0 citations
Due to their black-box nature, LLMs suffer from limited explain- ability and a lack of determinism. Their usage cost can also rise, particularly with repetitive tasks on large codebases. To mitigate this, we conduct a novel empirical study targeting three domain- specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep. We evaluate three LLMs (GPT-5.4, GPT-oss-120B, and Llama3.1-8B) on six diverse datasets covering four software- evolution tasks: API misuse correction, program repair, API migra- tion, and language version migration. Our results provide evidence that transformation rule synthesis moves beyond proof-of-concept with strong frontier models. GPT-5.4 achieves consistently high rule applicability rates and produces transformations closest to the ground truth across most benchmarks. Smaller and open-weight GPT-oss-120B and Llama3.1-8B models remain effective for simpler, localized changes but struggle with complex migration scenarios. We also observe non-negligible generalizability through the usage of meta-variables and through a high reuse score in the first quartile of many datasets. Finally, when compared to the anti-unification algorithm, LLMs outperform it in correctness, but underperform in rule applicability. Overall, our results show great potential for LLMs to generate sound, correct, generalizable, and reusable rules.
Axel Allain, Aymeric Blot, D. Khelladi et al.· 1 citation
Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead. We study if LLMs lean on priors or given semantics with a novel task--Program Executability Prediction (PrEx)--that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program's syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs. We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases. PrEx is available at https://github.com/EngineeringSoftware/prex.
Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa et al.· 0 citations
Large Language Models (LLMs) have shown promising performance in generating Object Constraint Language (OCL) constraints from natural language specifications. However, existing evaluations rely on publicly available UML models, which may overestimate generalization due to potential data leakage and reliance on recurring lexical and structural patterns. This paper introduces a transformation-based benchmarking approach for OCL generation, based on deterministic, semantics-preserving UML model transformations; such as identifier renaming, attribute reification, and association reification, that preserve specification intent while altering model representation. We evaluate a diverse set of closed-source and open-source LLMs under zero-shot, few-shot, and chain-of-thought prompting, measuring syntactic accuracy (well-formed and type-correct OCL) and semantic accuracy (correct interpretation with respect to the UML model). While leading models achieve high performance on non-transformed models (up to 92.10% syntactic accuracy and 74.56% semantic accuracy), performance degrades substantially under transformation, with best semantic accuracy dropping to 55.96% for closed-source models and 42.20% for open-source models. These results demonstrate that current LLMs remain brittle under semantically equivalent but structurally altered models, suggesting a sensitivity to surface-level patterns and highlighting the need for transformation-based benchmarks for evaluating OCL generation.
Hamza Attarwala, Moataz Chouchen, Omar Alam et al.· Proceedings of the ACM/IEEE...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
