EUE-Net: An Evidential Uncertainty Estimation-Based Dynamic Fusion Network for Vision–Radar Joint Gesture Recognition

Abstract

Gesture recognition is essential for human-computer interaction, with visual and radar sensing technologies extensively employed in their respective fields. The complementary nature of these modalities, characterized by their feature richness, spatial-temporal resolution, and environmental robustness, has inspired researchers to combine their strengths, enhancing overall discrimination capability. However, existing fusion algorithms assume both modalities are valid simultaneously. When one modality fails, e.g., visual sensing is compromised by poor lighting, it may introduce significant noise, potentially leading to unreliable recognition. To address this challenge, we propose an Evidential Uncertainty Estimation-based Dynamic Fusion Network (EUE-Net), which dynamically integrates information from both modalities at the evidence level by evaluating sample uncertainty, thereby enhancing the recognition reliability. Specifically, we first design a cross-modality feature calibration and integration module that leverages complementary information from individual modalities for feature calibration and integrates both to generate a pseudo joint-modality representation. Then, we model recognition probabilities distributions using a Dirichlet distribution and introduce Dempster-Shafer evidence theory to parameterize recognition probabilities and uncertainty as evidence, quantifying the reliability of the recognition results. By incorporating evidence from individual and joint modalities, credible decision fusion is performed. Experimental results demonstrate that EUE-Net outperforms state-of-the-art methods, particularly in scenarios where one modality fails.