LLM-Mestcase: A Test Case Modification Method Based on LLMs and Mutation Testing

Abstract

High-quality test cases are crucial for ensuring software reliability. However, automatically generating test cases that can precisely detect semantic defects remains challenging. Mutation testing, by evaluating a test suite's ability to detect artificially seeded faults (mutants), provides an effective benchmark for measuring and improving test quality. Nevertheless, leveraging the results of mutation testing, especially surviving mutants, to automatically guide test case generation and enhancement faces two major obstacles: 1) the semantic understanding gap-inferring high-level fault semantics from low-level syntactic variations of surviving mutants; and 2) the context integration challenge-generated test cases must align with the project's existing coding style, testing framework, and domain-specific knowledge. To address these challenges, this paper proposes an LLM-based, mutation testing-driven approach for test case generation. Our method first employs an Analysis Agent to perform semantic clustering and fault attribution on surviving mutants, identifying genuine fault patterns. Subsequently, a Generate Agent produces targeted test cases that are stylistically consistent and domain-relevant, leveraging the context of the original code and existing test suite. Finally, a Check Agent validates the effectiveness of the newly generated cases. We conducted experiments on a specialized code dataset from the coal industry. The results demonstrate that our approach achieves an accuracy of 84.41% in identifying the causes of surviving mutants, a passing rate of 81.71% in generated test cases, and eliminates 55.66% of the surviving mutants, significantly enhancing the fault detection capability of the test suite. This work paves a new path for intelligent test case enhancement by integrating the semantic understanding of large language models with the precise evaluation mechanism of mutation testing.