A Deep Biasing-Based Approach to Customizable Sign Language Recognition

Abstract

Sign Language Recognition (SLR) is a crucial technology for enabling social humanoid robots to interact with Deaf and Hard-of-Hearing (DHH) individuals. However, a critical capability gap exists in current SLR-equipped systems: models cannot recognize Out-of-Vocabulary (OOV) glosses – signs not seen during training -causing steep performance drops when deployed with real-world users whose vocabulary extends beyond the training set. This gap is particularly severe in human-robot interaction (HRI) contexts, where domain-specific terminology, proper names, or newly coined signs must be handled on-the-fly without costly system retraining. In this paper, we develop a customizable SLR method that enables robots to flexibly extend their sign vocabulary at inference time, without a significant training cost. The incapability of recognizing untrained signs involves two main factors: (1) the data scarcity of cheremes, the minimum units that compose sign language, leads to difficulties in defining new glosses, and (2) adding new glosses requires retraining the model with a significant cost. To address these issues, we propose automatic extraction of chereme-inspired sub-units via clustering for issue (1), and vocabulary extension based on Deep Biasing for issue (2). Experiments show that the proposed method improved Word Correct (WC) for OOV glosses by 39.1 points while minimizing degradation on trained glosses to 3.2 points, demonstrating its effectiveness for robust and customizable sign language interaction in real-world deployment.