Three-dimensional Pose Refinement with Spatial-Temporal Transformer Mechanism

Abstract

Deep learning-based human pose estimation extracts the human skeleton from a series of motions for computer vision applications, including rehabilitation, gaming, and sports analysis. The challenges of 2D human pose estimation lead to poor joint estimation due to depth ambiguity and joint occlusion. Current methods focus on understanding joint-to-joint relationships within a single frame. This approach ignores the temporal correlation of the joint across multiple frames, important for reconstructing the occluded joint. A spatial-temporal transformer-based refinement pipeline incorporating a depth map is proposed to address the limitations of self-occlusion. The spatial transformer models the joint relationships within the frame to capture the body's anatomical structure. While the temporal transformer models the joint movement across consecutive frames to refine the occluded joint. The depth map provides depth information for the relevant joint estimated from the RGB frames. Depth ambiguity can be reduced by understanding the exact depth of each joint. Experiments on the ITOP dataset demonstrate that the proposed method achieves mAP of 82.85% and an mDE of 7.12 cm, outperforming the baseline by 2.99% and 7.65%, respectively. The results of pose estimation using the spatial and temporal features can improve the location of human joint.