Agent-as-Director: Feedback-Driven Graph Propagation for Dual-Loop Real-Time Semantic Decision-Making in UAVs

Abstract

Multimodal large language models (VLMs) give unmanned aerial vehicles (UAVs) strong scene understanding, but their multi-second inference latency makes them unsuitable as synchronous decision makers. This paper proposes the Agent-as-Director dual-loop architecture, which removes the VLM from the decision-critical path: instead of emitting actions, the VLM asynchronously edits the weight matrix of a seven-node weighted fully-connected action graph (WFCAG), while a fast branch decides each frame through millisecond-level graph propagation over perceptual evidence, with a deterministic safety gating layer as the final arbiter. Causal and scoring feedback drive bounded online adaptation of the graph weights, with versioned rollback. In 100 search-and-approach flights in AirSim, the system achieves an 83% success rate, 331 ms average decision latency, and zero collisions, while the synchronous-VLM baseline Lance attains only 41% success with 37 collisions. The architecture decouples semantic understanding from real-time control, giving UAVs a low-latency, interpretable, and safe semantic decision-making scheme. This offers a practical route for injecting large multimodal models into latency-critical robotic systems, supporting applications such as autonomous inspection and cooperative engagement.