Hybrid Surface-EMG and EEG Fusion for Multi-Class Hand Gesture Recognition: A Comparative Study Under Cross-Subject and Within-Subject Evaluation

Abstract

Reliable hand gesture recognition plays an important role in human-computer interaction, prosthetic control, and rehabilitation systems. Surface electromyography (sEMG) has been widely used for gesture recognition because it directly captures muscle activity, whereas electroencephalography (EEG) provides complementary information related to motor intention. This work investigates whether combining both modalities improves classification performance under realistic evaluation settings. A reproducible framework is developed using synchronized eight-channel sEMG and EEG recordings collected from nine participants performing seven different hand gestures. After signal preprocessing, per-subject normalization, and leakagefree window-based segmentation ($\approx {19,300}$ epochs), both traditional machine learning algorithms and a dual-branch deep learning fusion model are evaluated under subject-dependent and subject-independent protocols. Experimental results show that the proposed fusion model reaches 88.3% accuracy (macro-$\left.F_{1}=0.88\right)$ under within-subject evaluation but falls to 30.5% on unseen subjects, quantifying the substantial subject-shift that limits out-of-the-box deployment. Notably, a gradient-boosted classifier over handcrafted features actually matches the deep model within-subject. Furthermore, it surpasses it on unseen subjects, indicating that the subject shift gap is the binding constraint rather than the model capacity. The modality ablation further shows that sEMG carries most of the discriminative information, while EEG contributes complementary features only after subject-specific calibration.