Code Transformation Rule Synthesis using LLMs: Potential and Limits
TL;DR
A novel empirical study targeting three domain- specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep, which shows great potential for LLMs to generate sound, correct, generalizable, and reusable rules.
Abstract
Due to their black-box nature, LLMs suffer from limited explain- ability and a lack of determinism. Their usage cost can also rise, particularly with repetitive tasks on large codebases. Code transfor- mation rules mitigate those challenges. This paper fills this gap, with a novel empirical study targeting three domain-specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep We evaluate three LLMs (GPT-5.4, GPT-oss-120B, and Llama3.1-8B) on six diverse datasets covering four software-evolution tasks: API misuse correction, program repair, API migration, and language ver- sion migration. Our results provide evidence that transformation rule synthesis moves beyond proof-of-concept with strong frontier models. GPT-5.4 achieves consistently high rule applicability rates and produces transformations closest to the ground truth across most benchmarks. Smaller and open-weight GPT-oss-120B and Llama3.1-8B models remain effective for simpler, localized changes but struggle with complex migration scenarios. We also observe non-negligible generalizability through the usage of meta-variables and through a high reuse score in the first quartile of many datasets. Finally, when compared to the anti-unification algorithm, LLMs outperform it in correctness, but underperform in rule applicabil- ity. Overall, our results show great potential for LLMs to generate sound, correct, generalizable, and reusable rules.
