Siamese masked reconstruction with temporal rotation consistency for inertial-based human activity recognition

Abstract

Inertial-based Human Activity Recognition (HAR) aims at inferring activities performed by an individual from data on wearable inertial sensors. However, HAR often suffers from poor generalization to unseen users due to data heterogeneity across different individuals and conditions. While collecting and curating large labeled datasets could help alleviate this limitation, it is also costly. To reduce this gap without relying on extensive annotated data, we propose a self-supervised framework grounded in how IMU signals transform under rotation: applying a rotation matrix changes how motion projects onto the sensor axes without altering the underlying motion. A model trained to align representations of a signal and its rotated counterpart is therefore encouraged to capture motion content rather than sensor-frame patterns. We exploit this property by training a Siamese masked convolutional autoencoder that learns by reconstructing masked inputs while aligning representations of original signals and its rotated version, further regularized by a temporal consistency loss that enforces agreement between the temporal structure of original and rotated representations within each window. We evaluate on four public HAR benchmarks covering diverse scenarios, including activities of the daily living (ADL) and sports activities, using cross-subject evaluation, where our method yields improvements of +1.4 and +1.2 percentage points over the strongest baseline with linear and MLP probes respectively, averaged across datasets and sensor positions. We further show that our method obtains competitive results against the fully supervised baseline in low-data regimes. The code is available on https://github.com/FranciscoCalatrava/Rotation_Siamese_Masked.git.