From Landmarks to Gestures: Real-Time Lightweight Gesture Recognition at the Edge

Abstract

The paper presents a real-time landmark-based pipeline for two-hand static gesture recognition targeting smart glasses applications. The proposed system extends a MediaPipe-inspired hand landmark extraction module with a lightweight multilayer perceptron classifier operating on a fixed two-hand landmark representation, improving recognition accuracy with negligible additional computational overhead. The proposed algorithm recognizes 33 HaGRIDv2 static gesture classes, including both single-hand and two-hand gestures, from hand landmark features rather than full RGB images. On HaGRIDv2, the proposed pipeline achieves a mAP of 93.0% and a macro F1-score of 92.0%. Compared with representative object-detection-based gesture recognition models, the proposed pipeline improves the best baseline mAP by 3.6 percentage points, while keeping the number of parameters in the same order of magnitude as lightweight detectors. The Gesture MLP itself requires only 0.021 M parameters and 0.13 ms of inference time. The complete system is deployed on an STM32N6570 processor, which integrates an ARM Cortex-M55 core operating at 800 MHz and an Neural Processing Unit (NPU) operating at 1 GHz. Experimental evaluations report runtimes from 17.97 ms in one-hand tracking mode to 113.85 ms in two-hand full-recognition mode. The Palm Detector is identified as the dominant latency component, mainly because it is only partially accelerated and includes software and hybrid CPU–NPU execution steps. Battery-life estimates, computed for a 175 mAh, 3.7 V Li-ion battery at 5 fps, show that hybrid full-recognition/tracking schedules can support several hours of operation, reaching approximately 18 h in the most favorable one-hand configuration. Overall, the results demonstrate that a compact landmark-based approach can achieve state-of-the-art recognition performance on 33 static gesture classes, while enabling real-time execution and energy-efficient operation suitable for long-lasting deployment on battery-powered smart glasses.