FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration

Abstract

AI coding assistants are rapidly becoming integral to modern software development. A key challenge in this space is the continual need to migrate and modernize codebases in response to evolving software ecosystems. Traditionally, such migrations have relied on rule-based systems and human intervention. With the advent of powerful large language models (LLMs), AI-driven agentic frameworks offer a promising alternative-but their effectiveness has not been systematically evaluated. In this paper, we introduce \FreshBrew{}\footnote{\url{https://github.com/mrcabbage972/freshbrew}}, a novel benchmark for evaluating AI agents on project-level Java migrations, with a specific focus on measuring an agent's ability to preserve program semantics and avoid reward hacking, which we argue requires projects with high test coverage for a rigorous and reliable evaluation. We benchmark several state-of-the-art LLMs, and compare their performance against established rule-based tools. Our evaluation on a challenging subset of 228 public Maven repositories that build on JDK 8, fail on JDK 17, and have high test coverage shows that the top-performing model, Gemini 2.5 Flash, can successfully migrate 52.3\% of projects to JDK 17. Our empirical analysis reveals novel insights into the critical strengths and limitations of current agentic approaches, offering actionable insights into their real-world applicability. Our empirical study reveals failure modes of current AI agents in realistic Java modernization tasks, providing a foundation for evaluating trustworthy code-migration systems. By releasing \FreshBrew{}, we aim to facilitate rigorous, reproducible evaluation and catalyze progress in AI-driven codebase modernization.