AutoQA-MAS: An LLM-Based Multi-agent System for Autonomous Website End-to-End Testing

Abstract

Manual execution of website end-to-end (E2E) test cases is a repetitive, labour-intensive process that creates bottlenecks in modern software development workflows. We investigate the feasibility of automating this process using AutoQA-MAS, a large language model (LLM)-based multi-agent system (MAS) comprising four specialised agents coordinated through a shared data pool: a Web Agent that interprets natural language test cases and interacts with the browser via Document Object Model (DOM) analysis; a Vision Agent that inspects screenshots and agent context for visual defects; an Accessibility Agent that audits Web Content Accessibility Guidelines (WCAG) 2.1 Level AA compliance; and a Report Agent that semantically deduplicates findings into a consolidated quality assurance (QA) report. The system was evaluated across 90 test case executions on a controlled e-commerce environment, 89 of which produced valid reports, comparing OpenAI GPT-5-mini, Google Gemini 2.5 Flash, and Anthropic Claude Sonnet 4.6. The system achieved an overall goal completion rate of 85.4%, bug recall of 76.9%, and accessibility issue precision of 76.0% (GPT-5-mini and Gemini only), with Claude Sonnet 4.6 achieving the highest bug detection performance (91.5% precision, 84.3% recall, 87.9% F1) while consuming 28% fewer tokens than GPT-5-mini. We observe that well-defined detection criteria, such as WCAG standards, yield significantly higher precision than loosely scoped bug detection (76.0% vs. 65.9%), motivating richer contextual prompting as a key direction for improvement. These results demonstrate the viability of LLM-based MAS for autonomous web QA, offering a scalable pathway to reduce manual testing effort, lower quality assurance costs, and accelerate software delivery cycles without proportional increases in engineering headcount.