A Preliminary Study on How AI Coding Agents Describe Refactoring Activities

Abstract

Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.