Deployment-oriented evaluation of large language models for Arabic emotion classification

Abstract

Large Language Models (LLMs) are increasingly used for emotion classification, yet Arabic evaluations remain limited and typically emphasise predictive performance alone. This study presents a deployment-oriented evaluation framework for Arabic emotion classification that complements conventional accuracy-based assessment with measures of output validity, operationalised through coverage and unknown rate, together with strict versus covered evaluation. The evaluation is conducted on 1002 manually annotated Arabic sentences spanning five emotion categories, with very high inter-annotator agreement ( \(\kappa = 0.9722\) ). We evaluate multiple proprietary and open-weight LLMs under a harmonised zero-shot setting and further examine the effects of few-shot prompting, three-vote stochastic aggregation, and run-to-run stability. Supervised Arabic encoder baselines, AraBERT and MARBERT, are additionally included as contextual task-specific reference models. The results show that the highest-performing LLMs combine strong predictive performance with high output validity, while differences in non-evaluable outputs are model- and protocol-specific. Few-shot prompting improves performance for most, but not all, evaluated models, whereas three-vote aggregation produces inconsistent effects across systems. Overall, the findings demonstrate that deployment-oriented indicators provide complementary information beyond predictive performance and support a more transparent evaluation of Arabic emotion classification systems under controlled inference conditions.