A Deep Learning based dual arm neural network for multi modal Indian sign language recognition using sEMG and motion sensors

Abstract

Sign language recognition plays a crucial role in enabling inclusive human–computer interaction, yet accurate recognition of complex gestures remains challenging due to the variability in motion dynamics and muscle activations. In this study, we propose a deep learning-based dual-arm neural network for recognizing Indian Sign Language gestures using multi-modal sensor data. Surface electromyography (sEMG), accelerometer, and gyroscope signals were collected from both arms of participants using the Delsys Trigno system across 50 distinct gestures. The proposed architecture processes left- and right-arm signals in parallel branches before feature fusion, enabling the model to capture complementary neuromuscular and kinematic patterns. Comparative experiments with single-arm baselines and alternative deep models, including LSTM and CNN-based networks, demonstrate that the dual-arm architecture consistently achieves superior accuracy and robustness. The results highlight the importance of multi-modal sensor fusion and deep learning in advancing automatic sign language recognition systems, paving the way for more reliable assistive technologies in communication accessibility.