A Few-shot Learning Approach for Dynamic Arabic Sign Language Recognition

Abstract

Sign language's importance comes from its vital role in improving the lifestyle of deaf and hearing-impaired people.Despite the importance of Arabic Sign Language (ArSL) in playing this vital role in the Arab world, ArSL recognition researches are still limited.This limitation is largely due to the lack of high-quality, large datasets needed to train deep learning models, as well as the challenges associated with the signer-independent mode, which requires the model to generalize to new signers it has not seen during training.In this paper, we propose an approach to recognize isolated dynamic Arabic signs, evaluating the model's performance in both signer-dependent and signerindependent modes, with a particular focus on improving the ability to generalize in the signer-independent mode.Initially keypoints were extracted from sign videos using the MediaPipe library.The proposed approach is based on prototypical network, which is one of Few-Shot learning techniques that enables the model to learn and generalize from a limited number of samples, reducing the need to collect huge amount of data.This prototypical network depends on a hybrid encoder that combines CNN to extract spatial features from keypoints and Transformer to extract temporal features.The model was trained using an episodic training method with N-way K-shot strategy, which simulates real testing scenarios in Few-Shot learning environments.After training, the model was tested using episodic testing also with the N-way K-shot strategy and achieved an accuracy of 98.91% and 95.89% in signer-dependent and signerindependent modes, respectively, which is an achievement that confirms the effectiveness of the proposed approach in addressing the challenges of data scarcity and the difficulty of generalization to new signers.