Photoplethysmography and vital sign measurement using 3D-multispectral multimodal imaging

Abstract

Contactless vital-sign monitoring can improve comfort and hygiene in long-term, sleep, home, and clinical monitoring, but robust estimation of heart rate (HR), respiratory rate (RR), and oxygen saturation (SpO2) remains a coupled sensing and algorithmic problem. This paper summarizes our research line on remote photoplethysmography and vital sign measurement using 3D-multispectral multimodal imaging. We first review the CoViTal system, which integrates RGB, NIR, thermal, and 3D imaging for registered multimodal acquisition and 3Dsupported ROI tracking. We then describe a miniaturized multispectral multi-aperture camera (MAC) as a compact extension toward portable and home-oriented monitoring. Based on these systems, we organize the preprocessing pipeline, HR estimation from multispectral rPPG, RR estimation from thermal and rPPG signals, SpO2 estimation with spatiotemporal learning, multi-task learning, and domain-adaptive oximetry. Clinical validation with PSG reference showed second-level MAEs of 2.17 bpm for HR, 2.09 bpm for RR, and 1.38% for SpO2 during oxygen desaturation events. MAC-based validation further demonstrated feasible RR, HR, and SpO2 estimation with a trade-off setup. These results indicate that multispectral imaging can support both clinical monitoring and future home-oriented deployment, although challenges remain in motion robustness, clinical data coverage, cross-domain generalization, and real-time operation.