SIGN-AIR: real-time sign language recognition and translation using gesture-aware neural networks on aerial robotic platforms

Abstract

Abstract Communication barriers experienced by deaf and hard-of-hearing individuals remain a significant challenge, particularly in situations where conventional communication infrastructure is unavailable or fixed-camera systems have limited coverage. This paper presents SIGN-AIR, a UAV-assisted framework for real-time sign language recognition and bidirectional translation. Unlike conventional approaches based on fixed cameras, wearable sensors, or depth cameras, the proposed framework integrates a drone-mounted stabilised RGB camera with onboard edge computing to enable flexible gesture recognition in indoor and outdoor environments. The system combines MediaPipe-based hand landmark extraction with a hybrid CNN–BiLSTM architecture to model both the spatial and temporal characteristics of American Sign Language gestures. Experimental evaluation on a combined dataset comprising the ASL Alphabet and WLASL achieved an overall recognition accuracy of 98.63%, together with high precision, high recall, and a median inference latency of less than 2 s. These results demonstrate the technical feasibility of the proposed UAV-assisted recognition framework under the evaluated experimental conditions and indicate its potential to support communication in scenarios where fixed-camera systems may be affected by occlusions, limited fields of view, or environmental variability. Furthermore, the proposed bidirectional translation pipeline has the potential to facilitate both sign-to-speech and speech-to-sign communication for prospective applications in accessibility, education, public service interactions, and emergency response. Although the current study demonstrates promising technical performance, comprehensive real-world field validation under representative operational conditions remains an important direction for future work.