A lightweight multi-branch transformer architecture for Kazakh Sign Language recognition under signer-independent conditions

Abstract

Sign Language Recognition (SLR) in low-resource sign languages is still limited due to the lack of annotated data, high variability of signers, and lack of benchmarks; Kazakh Sign Language (KzSL) is a representative example of such a low-resource language. The proposed research introduces a novel method for isolated KzSL recognition based on a purpose-collected dataset of RGB video streams of five signers, represented by 1662-dimensional per-frame pose, facial, and hand landmark vectors extracted using MediaPipe Holistic. The final dataset consists of 39,979 gesture samples across 187 output classes (186 gesture classes and 1 no_event class), each of which is resampled into a temporal landmark tensor 56 × 1662. A lightweight model, Transformer model (SignTransformerModel, 3,529,272 trainable parameters), is proposed with two parts: (1) Each landmark is projected through a separate branch of a MultiBranchStem, a branch representation is obtained for each landmark and fused into a unified representation of 256 dimension; (2) the fused sequence is processed by a four-layer Transformer encoder with eight attention heads, aggregated into a single sequence representation by a TemporalAttentionPool, which combines attention and max pooling, and then passed to a classification layer. The model was compared against three baseline architectures—a Standard Transformer, a Temporal Convolutional Network, and a Bidirectional GRU—and obtained a mean accuracy of 0.6698 ± 0.0376, a mean macro F1 of 0.4207 ± 0.0420, and a mean weighted F1 of 0.6546 ± 0.0375 over five outer test partitions of a nested, signer-independent Leave-One-Signer-Out cross-validation, outperforming all three baselines in macro F1 on every partition. The results show that compact landmark-based Transformer models are feasible for the recognition of KzSL, though aggregate accuracy is substantially inflated by the dominant no_event class (gesture-only accuracy: 0.4490), and that inter-signer variability and incomplete class coverage by signers as well as recurrent class confusions exist and present clear directions for expanding the dataset and fine-tuning the model with expert guidance.