FER20E: An Extended Facial Expression Recognition Dataset With 20 Discrete Emotions

Abstract

Facial emotion recognition (FER) has traditionally focused on a limited set of basic expressions, often failing to capture the complexity, subtlety, and cultural variability of real-world human emotions. To address these limitations, this paper introduces FER20E, a large-scale facial expression dataset comprising 20 emotion categories, including both basic and fine-grained affective states. The proposed emotion taxonomy is systematically derived by integrating facial action coding system (FACS)-based action units (AUs) with the valence-arousal circumplex model, ensuring both interpretability and psychological validity. To enable scalable and reliable annotation, we develop a data annotation tool (DL-DAT) that follows a semi-automated, human-in-the-loop pipeline. To validate the effectiveness and relevance of FER20E, we conduct extensive experiments using recent state-of-the-art models, including convolutional neural networks and transformer-based models. Results demonstrate that lightweight models such as MobileNetV2 and SqueezeNet achieve competitive performance while incurring significantly lower computational cost, enabling real-time deployment. Furthermore, transformer-based models with large-scale pretraining (ViT21k) achieve superior recognition accuracy, highlighting the importance of representation learning. Additional analysis reveals challenges related to emotion ambiguity, overlapping AUs, and cross-cultural variations, underscoring the need for fine-grained, robust FER systems. The FER20E dataset provides a comprehensive benchmark for advancing emotion recognition in unconstrained and real-world scenarios. The dataset and implementation details will be publicly available at https://github.com/akstheme/FER20E