Dual-stream attention and feature fusion for human activity recognition via wearable sensors

Abstract

Multi-sensor-based human activity recognition (HAR) has attracted considerable attention in recent years. Despite significant advancements achieved through deep learning, extracting refined and subtle features from multiple sensors placed on different parts of the body remains a challenge. Accurate sensor modality attention and temporal information processing are crucial for enhancing model performance. In this paper, we propose a new Temporal and Sensor-Modality Attention with Feature Fusion Network (TSFF). TSFF consists of two modules: the Temporal and Sensor Modality Attention block (TS) and the Global and Local Feature Fusion block (GLFF). The TS module extracts sensor modality attention and temporal attention in parallel using two branches, while the GLFF module dynamically fuses features from the sensor modality attention branch and the temporal stream through global and local attention mechanisms. We evaluated TSFF on three widely used public datasets–PAMAP2, Opportunity, and WISDM. When integrated with the ResNet backbone, TSFF achieved accuracies of 94.64%, 92.18%, and 97.96% on these datasets, respectively. Ablation studies further indicate that jointly modelling temporal and sensor-modality information with adaptive global–local fusion improves performance over the corresponding backbone under the evaluated settings. Visual analyses additionally provide qualitative evidence that the learned representations exhibit clearer class separation and differentiated weighting patterns across sensors and time steps.