MuMuTestUp: Mutation-Based Multi-agent Test Case Update

Abstract

Modern software systems evolve rapidly under continuous integration and deployment (CI/CD) practices, in which tests act as critical gatekeepers of software quality. However, when substantial code changes are introduced, existing test cases may become obsolete, leading to compilation failures, erroneous test behaviors, or inadequate regression coverage. Such issues can disrupt CI/CD pipelines, degrade development productivity, and ultimately undermine overall software quality. Many efforts are devoted to designing automatic test case update methods to address these issues. The most recent approaches rely on large language models (LLMs) to iteratively refine test cases using execution feedback from compilation errors or coverage reports, and on context retrieved via exact-matching approaches. They also prioritize test executability and line coverage to quickly build executable, correct test cases from the original broken test cases. Despite their correctness, current approaches face three limitations: (1) they focus on executabilty but overlook the adequacy of test assertions, which lowers the capability of test cases to detect faults; (2) they utilize only coarse line coverage singals instead of specific information about uncovered lines and branches; (3) they use exact-matching context retrieval approaches, which fails to provide accurate context given potential hallucinated queries from LLMs. To address these challenges, we propose MuMuTestUp, a Mutation-guided, Multi-agent framework for automated test case updating. MuMuTestUp integrates three specialized agents: (1) a Mutation Analysis agent that leverages surviving mutants as indicators of weak or missing test assertions and generates individual repair instructions to strengthen or synthesize assertions for each surviving mutant, (2) a Coverage Analysis agent generates individual repair instructions for each uncovered line, uncovered branch rather than exposing raw coverage signals to the LLM, and (3) a Semantic Retrieval agent that uses semantic-similarity search to handle unavailable or hallucinated symbols. Additionally, we construct Prbench, a pull-request–level dataset of 571 samples from 10 open-source Java projects that considered cross-commit update scenarios, validated through three rounds of execution following prior studies to detect outdated tests. We evaluate MuMuTestUp against state-of-the-art baselines using both open-source and closed-source LLMs (Deepseek-V3.2 and GPT-4.1). With GPT-4.1, MuMuTestUp achieves a line coverage of 88.94%, branch coverage of 63.36%, and mutation score of 72.39%, outperforming the best baseline by 5.33%, 19.93%, and 16.66%, respectively.