GRAFT-SLR: Graph-Aware Region Attention and Flow Token Transformer for Isolated Sign Language Recognition

Abstract

Isolated Sign Language Recognition (ISLR) requires an effective representation of both fine-grained hand articulations and the complex spatio-temporal motion patterns that define each sign. Existing approaches often rely solely on RGB sequences, optical flow, or skeletal keypoints, which limits their robustness under occlusion, motion blur, and signer variability. To address these challenges, we propose a Graph-aware Region Attention and Flow-Token Transformer (GRAFT-SLR), a novel cross-modal framework that integrates discrete motion tokens with structured skeleton graph representations for isolated sign recognition. The proposed model introduces a motion-vector-quantized codebook that converts short RGB–flow segments into compact, discrete motion tokens, enabling symbolic modeling of micro-gesture dynamics. In parallel, a spatio-temporal skeleton graph encoder extracts structural pose features enriched with a lightweight wrist–knuckle contour descriptor. A gated cross-modal transformer fuses the two modalities, in which attention is adaptively weighted to enhance robustness to noisy or missing keypoints. Furthermore, GRAFT-SLR employs a dual-stage training strategy that combines contrastive motion-structure alignment and masked token prediction to learn generalized gesture primitives prior to supervised fine-tuning. Extensive experiments on benchmark datasets, including Include-50 and WLASL-100, demonstrate that GRAFT-SLR achieves superior accuracy compared with state-of-the-art RGB, optical-flow, and skeleton-based baselines. Ablation studies validate the contribution of discrete motion tokens, gated cross-attention, and contour-enhanced skeleton features. The results confirm that GRAFT-SLR provides a robust, efficient, and interpretable solution for isolated SLR, while offering a scalable foundation for future multi-modal sign language understanding.