Self-Supervised Dual-Stream Temporal Learning for Neonatal Pain Estimation

Abstract

Neonatal pain estimation from video is clinically important but challenging due to limited labelled data and the reliance of existing automated methods on facial appearance alone. Current approaches use handcrafted features or single-frame deep representations, overlooking the temporal body dynamics (such as limb flexion, postural tension, and rhythmic movement) that clinicians routinely use alongside facial cues. We propose a self-supervised dual-stream temporal framework that combines facial and body information for framewise pain-score regression on APN-db. A face encoder is pretrained via contrastive learning on unlabelled neonatal video, producing domain-adapted features that substantially outperform generic pretrained representations. These frozen features, together with body-pose keypoints, are passed through separate temporal transformer encoders over short windows before learned gated fusion. This design captures body-pain dynamics that are invisible in single frames while leveraging unlabelled data to overcome the label-scarcity bottleneck. We evaluate under leave-one-subject-out cross-validation on two dataset splits and demonstrate substantial improvements over the prior APN-db benchmark, reducing mean absolute error by \(35\%\) on our primary split and \(25\%\) on a like-for-like reconstruction of the benchmark subset. A complementary experiment using a movement-derived target further validates that the dual-stream architecture effectively fuses body information when the prediction target depends on it.