Understanding Tool-Mediated Interaction Failures in Agentic GUI Software: An Empirical Study
Large language model (LLM) agents increasingly complete tasks through live graphical user interfaces (GUIs). Because the model cannot directly operate the GUI, these systems often rely on tools to capture screenshots or UI trees, translate model outputs into GUI actions, and report whether the interface changed. We stu...