A Survey on Reinforcement Learning Optimization Methods for Multi-Agent Collaboration of Large Language Models

Abstract

The integration of reinforcement learning (RL) into the optimization of multi-agent collaboration for Large Language Models (LLMs) is an important combination of two advanced areas, Multi-Agent Systems (MAS) and LLMs. This paper thoroughly examines the main approaches, evaluation standards, recent progress and existing problems in this new field. The research categorizes the existing solutions into three types. The first type is the individual reward-based fine-tuning which optimizes each agent individually. The second one is the joint reward-based centralized training which uses a common reward signal for collaborative learning. The third type is the interaction during the inference phase which enhances cooperation through real-time communication during the generation process. Among these, the joint reward methods demonstrate greater efficiency and effectiveness in cooperation, but they are still restricted by the difficulty in data generalization and model heterogeneity. After careful analysis of these problems, this paper proposes two possible directions for further research. One is to establish dynamic evaluation criteria to improve the evaluation accuracy, and the other is to adapt to various model architectures. The paper's objective is to provide a comprehensive guide to assist future studies on RL-supported multi-agent LLM collaboration.