Multi-Agent Reinforcement Learning with Intrinsic Motivation for Cooperative Navigation

Abstract

This paper investigates the challenges of achieving robust cooperative navigation in multi-agent reinforcement learning (MARL) scenarios, addressing the limitations of relying solely on extrinsic rewards. We propose a novel framework integrating intrinsic motivation – specifically, curiosity and surprise – into each agent's reward function. Our core claim is that extrinsic rewards alone frequently fail to foster genuine cooperation, leading to suboptimal collective behavior. By introducing intrinsic motivation, agents are driven to explore and interact with their environment in ways that inherently benefit the overall task, promoting more effective coordination. We define a mathematical framework for this approach, outlining the agent reward function and the interaction dynamics. The key equation representing the agent's reward is: Ri(si, ai, s'i) = Rext(si, ai, s'i) + λ * I(si) where: * Ri(si, ai, s'i) is the reward received by agent *i* at state *si*, taking action *ai* and transitioning to state *s'i*. * Rext(si, ai, s'i) is the extrinsic reward, representing the reward for completing the primary navigation task. * λ is a weighting factor controlling the influence of intrinsic motivation. * I(si) is the intrinsic motivation signal, representing the level of novelty or surprise in the new state *si*. The exploration strategy, driven by the intrinsic motivation signal, is then implemented using a stochastic policy, denoted as πi(ai | si). The overall objective is to minimize the expected discounted sum of rewards: J = Eπ[ Σi=1K Σsi, ai Ri(si, ai, s'i) ] We demonstrate the effectiveness of our approach through theoretical analysis and simulations, showing that incorporating intrinsic motivation significantly improves cooperative behavior compared to standard extrinsic reward-based MARL methods. Future work will explore different intrinsic motivation mechanisms and their optimal weighting.