Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation

Abstract

Software testing is a critical, yet resource-intensive phase of the software development lifecycle. Search-based approaches typically achieve high coverage but produce tests with low readability, whereas large language model (LLM)-based methods generate more human-readable tests but often suffer from low coverage and compilability. While the majority of research efforts have focused on improving test coverage and readability, comparatively less attention has been paid to enhancing the robustness of bug detection. To address this gap, we propose AdverTest, a novel adversarial framework for LLM-powered test case generation that pairs a test case generation agent T with a mutant generation agent M: M persistently creates mutants "hacking" the blind spots of T's current test suite, whileT iteratively refines its tests to "kill" the challenging mutants, with the interaction guided by both coverage and mutation scores. Experimental results on Defects4J show that our approach improves fault detection rates by 8.56% over the best existing LLM-based methods and by 50.20% over EvoSuite, while remaining competitive on line and branch coverage.