Gesture Control Automation: A Webcam-Based Assistive System for Communication and Computer Control

Abstract

Gesture recognition is becoming a useful way of interacting with computers without using traditional devices such as keyboards, mice, and touchscreens. This type of interaction can be especially helpful for people who have physical or communication difficulties. In this paper, we propose Gesture Control Automation, a webcam-based system that recognizes custom hand gestures and uses them for communication and computer control. The system captures live video through a webcam and uses OpenCV for image processing and MediaPipe for hand landmark detection. The detected landmarks are used to identify predefined hand gestures. The main idea of the proposed system is to use the recognized gesture for more than one purpose. A gesture can be converted into text, the text can be converted into speech, and selected gestures can also be used to perform predefined computer operations. To reduce accidental commands, the system can use confidence checking and multiple-frame confirmation before executing an automation action. The proposed approach is simple, low-cost, and does not require special wearable sensors. The system is mainly intended to improve accessibility and provide a common interface for communication and computer control. The work also identifies a gap in existing systems, where gesture recognition, communication, and automation are often developed separately. The proposed system attempts to combine these Keywords— Gesture Recognition; Computer Vision; MediaPipe; OpenCV; Assistive Technology; Automation