LF-Former: A Multi-Agent Policy With Ordered Decision Structures for Multi-Arm Manipulation

Abstract

Imitation learning has shown strong performance in robotic manipulation. Extending it to multi-arm settings, however, remains challenging because coordination becomes increasingly difficult as the number of arms grows. To address this, we propose the leader-follower transformer (LF-Former), a multi-agent policy architecture that leverages an ordered decision structure for multi-arm coordination. Under a given ordering, agents predict actions sequentially, with each agent conditioned on preceding agents' action chunks and shared temporal context that summarizes past observations across all agents. LF-Former adaptively integrates these inter-agent signals through a gated group attention mechanism. Furthermore, rather than manually fixing a decision ordering, we formulate ordering selection as a multi-armed bandit problem and identify an effective decision ordering during training. Experiments on simulated and real-world multi-arm manipulation tasks show that LF-Former consistently outperforms decentralized policies and matches or exceeds centralized baselines, especially in tightly coupled coordination scenarios. We further show that bandit-based ordering selection effectively identifies strong decision orderings, often matching or exceeding the best fixed ordering without requiring the oracle choice.