Autonomous Refactoring for Perfective Maintenance at Scale: The Agentic RACER System and its Empirical Evaluations

Abstract

Automating software development is an important aspect of software engineering. Generative Artificial Intelligence (GenAI) Agents that control multiple tools and use feedback with some autonomy to iteratively solve problems are emerging as promising tools for increased software production. To achieve autonomous software maintenance with minimal human involvement, such agents must not only solve human-posed problems but also decide which problems to address. We report our experiences developing a GenAI agent that autonomously formulates, prioritizes, and completes a limited set of code improvement tasks enterprise-wide, and we evaluate this agent from two perspectives: first, developing a methodology to estimate effort savings from such autonomous agents by comparing time spent on agent-introduced tasks and agent maintenance efforts against manual execution; and second, comparing the impact of manual versus agent-driven code improvements on future codebase maintenance. Our results demonstrate significant effort savings, even after accounting for effort in agent development, and a Difference-in-Differences (DiD) analysis found that automated improvements had an effect on the reduction of future maintenance effort that was as strong as or stronger than manual improvements confirming the feasibility of autonomous maintenance. While the relaxed completion and timeliness needs of perfective maintenance make it an ideal candidate for autonomy, a degree of autonomy may be achievable in other areas of software development as well.