MERA: A Multi-expert LLM Framework for Severity Assessment of User Tickets in Mobile OS Beta Testing

Abstract

Large-scale mobile OS beta testing generates massive ticket vol- umes, creating a triage bottleneck in which assigning severity dic- tates engineering priority. Automating this is difficult: unlike server logs with rich telemetry, beta tickets rely on vague, symptom-driven user descriptions, which confound standard AIOps tools. Tradi- tional classifiers based on deep learning or pre-trained language models (PLMs) further struggle to adapt to the rapid concept drift triggered by frequent feature updates, and their black-box nature fails to provide the reasoning transparency that engineers require for trust. We observe that experienced triage engineers resolve these challenges by triangulating across user intent, system context, and historical precedent—a discipline that no single monolithic model captures. We present MERA (Multi-expert Evidence Reasoning and Arbitration), the first production-deployed multi-expert LLM framework for severity assessment in telemetry-void mobile OS beta testing. MERA operationalizes this discipline by decoupling severity assessment into three expert perspectives and fusing their evidence into an auditable decision. To meet strict online latency budgets, MERA distills the reasoning capability of a large teacher model into three lightweight experts, and a dynamic fusion layer weighs their evidence to produce a severity decision with human- reviewable rationales. In the mobile OS beta-testing pipeline of a leading global information and communications technology (ICT) provider, MERA serves as the default severity grader for all incom- ing beta tickets, with engineers performing spot-checks. Over five months it processed more than 370,000 tickets at 85% accuracy, sub- stantially outperforming baselines under concept drift and reducing manual triage effort by 60%.