TPA-Seq2Seq: Tri-Prior Aligned Sequence-to-Sequence Learning for Continuous Sign Language Recognition

Abstract

With the growing emphasis on accessibility-oriented technologies and inclusive intelligent systems, Continuous Sign Language Recognition (CSLR) has attracted increasing attention as a key technique for bridging communication between Deaf and hearing communities. However, existing methods still suffer from insufficient exploitation of visual information, weak temporal alignment supervision, and inconsistency between training and inference, making it difficult to jointly improve recognition accuracy and model robustness. To address these issues, we propose TPA-Seq2Seq (Tri-Prior Aligned Seq2Seq), a tri-prior enhanced Seq2Seq framework for character-level CSLR. Specifically, we introduce PGF (Pose-Guided Fusion) to extract hand-arm-face keypoint descriptors offline through the MediaPipe pipeline and fuse them with RGB-based temporal semantics in a lightweight manner, thereby supplementing fine-grained visual priors. We further design ATAL (Auxiliary Temporal Alignment Loss), a CTC-based auxiliary constraint imposed on the encoder side to strengthen explicit temporal alignment supervision. In addition, we propose SSF (Scheduled Semantic Forcing), a piecewise teacher-forcing decay strategy that alleviates exposure bias and improves generalization. Experiments on the CSL dataset show that TPA-Seq2Seq reduces WER compared with the ResNet18-LSTM baseline and maintains consistent performance across different random seeds.