Interrater reliability of software optimized movement assessment with traditional methods and video-based functional movement screen scoring and compensatory movement identification

Abstract

Introduction: The Functional Movement Screen (FMS) is a seven-part movement assessment used to identify injury risks caused by faulty biomechanics. There are three barriers in traditional FMS assessments that could affect the tool’s validity: the subjectivity of human scores that could cause bias, the need for in-person evaluations which limit access for remote patients, and the requirement of specialized training to use the tool, which makes it less accessible. This study investigates the effectiveness of SOMA, an AI-based web application that uses the MediaPipe framework to automatically assess (FMS) performances. Methods: This study employs a quantitative, cross-sectional, comparative design to evaluate SOMA’s interrater reliability in a single-point analysis. Thirty three active adult students from Silliman University participated in the study , with assessments conducted by SOMA, a traditional rater, and a virtual rater. Results: In comparing SOMA’s scores and identified compensatory movements with human raters, the study found that there was generally fair to almost perfect agreement (0.38 - 1.0) between SOMA and the human raters for the scores as measured by Gwet’s AC2, while agreement ranged from substantial disagreement to almost perfect agreement (-0.66 – 0.97) across different compensatory movements as measured by Gwet’s AC1. Results indicated that the reliability differences across FMS test scores, being almost perfect for some tests like the Overhead Squat and Push-Up Stability, while showing lesser reliability for others like Shoulder Mobility and Straight Leg Raise. Discussion: The results of this study show that the FMS scores obtained by SOMA have reliability with traditional face-to-face assessments and virtual video-based assessments that cannot be expected merely by chance, ranging from fair agreement to almost perfect agreement. Differences in scoring were thought to be due to methodological differences, perceptual limitations of raters, and SOMA’s scoring system which utilized rigid thresholds. Reliably identifying specific compensatory movements, which SOMA did with varying reliability with either rater, could be attributed to various factors such as advantages and disadvantages with the methods either rater used, subjectiveness in the FMS scoring criteria and compensatory movement definitions, and limitations in the MediaPipe library. Future studies could be done to assess SOMA’s validity compared to gold-standard technology and SOMA could be used to complement assessments performed by human raters of the FMS in the future.