A Multi-Agent LLM Framework for Automated Software Testing

Abstract

Research on integrating specialized, language-model-based agents for automated test-case generation remains limited, and the detection criteria under which such systems are evaluated have not been standardized, which makes reported rates difficult to compare across studies. This paper presents a multi-agent testing framework in which requirement analysis, test-case generation, sandboxed execution, and defect detection are delegated to four distinct agents, and in which detection rests on a composite rule combining the execution signal with the semantic judgment of a dedicated diagnostic agent. The framework is evaluated on the QuixBugs dataset of forty Python programs under three criteria of increasing strictness. A two-run differential protocol against the reference implementation, recommended here as the primary indicator, yields a detection rate of 55.0%, against 90.0% under the unconditional criterion conventional in prior work and 5.0% under a strict criterion requiring the suite to pass entirely on correct code. The thirty-five-point gap is traced to language-model test artifacts, of which hallucinated oracle values and signature mismatches account for the majority. An ablation with a same-model single-agent baseline attributes the framework’s advantage to role decomposition rather than to the diagnostic agent specifically, and the distribution of detection across the fourteen defect classes defined by the benchmark is reported descriptively.