Evaluating Coloured Blob Tracking, CNN and Posture Detection computer vision models on latency and accuracy in the application of a virtual drum kit

Abstract

Drumming is often inaccessible to beginners due to high costs, space constraints, and noise. While commercial "drumless" virtual systems exist, they rely on expensive hardware like VR goggles, infrared cameras, and motion-sensor drumsticks. These solutions remain costly and inconvenient, failing to address the core issue of accessibility. This paper proposes a low-cost computer vision (CV)-based virtual drum system that works on minimal hardware, such as a consumer laptop with a standard webcam.This study investigates the effectiveness of a range of computer vision techniques for real time drumstick and foot tracking, along with reliable event detection for a virtual drum kit. Four models were implemented using different combinations of established CV methods, including coloured blob tracking, Kalman filtering for motion prediction, dynamic region of interest (ROI) scaling, posture tracking using the MediaPipe pose estimation framework, and convolutional neural network (CNN) based drumstick detection using the YOLOv8 object detection model. All models were designed to track two drumsticks and two knees and detect the corresponding events. Each model was bound to identical hardware constraints. Performance evaluation in this study focuses on two primary metrics: accuracy, measured using precision, recall, and F1-score derived from false positives and false negatives produced during drumming sequences; and latency, measured as the temporal difference between the actual moment a drum hit occurs and when the system registers that event. These metrics were chosen because they directly impact the user experience in a real-time virtual musical instrument.