PFEA: a VLM-based high-level natural language planning and feedback embodied agent for human-centered AI

Abstract

The rapid advancement of Large Language Models (LLMs) has led to significant progress in Artificial Intelligence (AI), ushering in a new era of human-centered AI (HAI). Intelligent agents powered by LLMs provide new opportunities for realizing HAI. However, existing LLM-based embodied agents often lack online planning capabilities and may generate actions involving objects that are not present in the current environment. In this paper, we propose a closed-loop framework for planning and evaluation of a vision-language model-based robotic manipulation agent operating in tabletop object rearrangement and manipulation tasks. These tasks require a robot to interpret high-level natural language commands and perform multi-step actions such as stacking objects, sorting objects by category or attribute, placing objects into target containers, and organizing mixed desktop scenes. The proposed architecture consists of a human–robot speech interaction module, a vision–language agent module (including a planner, translator, and evaluator), and an action execution module. Within this framework, the vision–language planner decomposes high-level instructions into executable task steps via Chain-of-Objects Reasoning, enabling a deeper understanding of the visual environment, including spatial relationships and object attributes. Meanwhile, the task evaluator monitors execution outcomes and provides result-oriented feedback, allowing the system to replan when errors occur. Experimental results show that, compared with baseline methods, the proposed agent improves the average task success rate by approximately 20% in both simulation and real-world environments, significantly enhancing its ability to execute complex natural language instructions. This work demonstrates the potential of closed-loop vision-language planning for human-centered robotic manipulation. Code: https://github.com/subforpaper/PFEA.