Low-Latency Human-in-the-Loop Robotic Teleoperation With Control–Media Separation Using MQTT and WebRTC

Abstract

Low-latency teleoperation is a core requirement for Artificial Intelligence of Things (AIoT) robotics, yet achieving responsive closed-loop manipulation while streaming live video and running on-the-fly vision inference remains challenging under geographically distributed networks. This article presents a modular AIoT teleoperation framework for human-in-the-loop robotic manipulation that separates the control and media pipelines using message queuing telemetry transport (MQTT) for real-time command delivery and web real-time communication (WebRTC) (via LiveKit) for low-latency video streaming. Lightweight visual perception is enabled through a You Only Look Once version 11-nano (YOLOv11-nano) proof-of-concept assistive visual overlay that provides real-time object-level annotations to a cross-platform Flutter-based mobile interface during teleoperation. On the actuation side, control commands are transmitted to an ESP8266-based communication node and executed by an Arduino Mega2560 controller to coordinate multiaxis robotic arm motion. Backend services are deployed on cloud infrastructure to support scalable access across regions. The perception model is trained on a 32 GB workstation and achieves millisecond-level inference performance (1.5 ms inference-only, $\approx$11.5 ms end-to-end per frame). The trained model is further evaluated on lightweight 8 GB RAM nodes under VPN-enabled cross-border conditions, including routes to Hong Kong, Japan, and Belgium. Experimental results demonstrate actuator response latencies as low as 0.2 s in local networks and below 0.7 s in cross-border scenarios, confirming that the proposed framework maintains real-time responsiveness even on resource-constrained hardware. Beyond baseline system benchmarking, we enhance journal-grade reproducibility by reporting multirun training stability under controlled illumination variability using precision–recall curves and normalized confusion matrices. Furthermore, the framework supports nonintrusive extensibility, exemplified by an optional infrared gesture-control module that can be integrated without altering the perception–control decoupling or the underlying actuation interface. Overall, the work provides a standards-aligned reference design for human-in-the-loop AIoT teleoperation, while demonstrating how lightweight object-level perception can be integrated as an assistive visual overlay.