Broken time, stable models? Evaluating desynchronization robustness in wearable human activity recognition
Wearable-based human activity recognition (HAR) has emerged as a valuable method for capturing activities across diverse domains, including rehabilitation, occupational ergonomics, sports, and human-computer interaction (HCI). While recognition performance can be significantly enhanced by leveraging multiple complementary sensors, this approach requires accurately synchronized time bases across all devices. Although previous studies on synchronization in HAR suggested that sub-second accuracy is advisable while sub-100 ms accuracy is unnecessary, the specific effect of time discrepancies on machine learning models has, so far, remained unexplored. We address this gap by introducing an experimental paradigm for systematically evaluating the impact of time discrepancies in multi-wearable HAR, which we evaluated in two experiments. In our first experiment, we use the example of multi-stage temporal convolutional networks (MS-TCN) for sequence-to-sequence action segmentation, simulating the time discrepancies of time offset and clock skew via rational resampling. Our evaluation spanned 30,025 training and validation runs across different model configurations, totaling over one million core-hours of computation. Our results reveal that time offsets larger than 167 ms should be avoided in training datasets, and offsets beyond 333 ms can already significantly degrade HAR performance for typical activities of daily living (ADLs). Subsequently, we performed a second experiment focusing on the impact of time offsets on inference in models trained on synchronized datasets. Our evaluation spanned temporal convolutional networks, LSTMs, and Transformer architectures across five architectural configurations, each with two different temporal input lengths. The results indicate that LSTMs for action segmentation are more robust to desynchronization, while other architectures exhibited a marked performance degradation beyond desynchronization offsets spanning 167 ms. Our findings have implications for the design and deployment of multi-wearable HAR systems and may extend to other multi-sensor contexts.