MoE-Pointer: Seq2Seq Reinforcement Learning for Dynamic Multi-Echelon Pickup-and-Delivery with Courier-Drone Relay
TL;DR
MoE-Pointer is proposed, a unified reinforcement learning framework that reformulates DM-PDP into a sequence-to-sequence generation task and introduces a Prior-Guided Soft Mask to guide exploration within the exponentially large action space.
Abstract
Hybrid delivery systems integrating couriers and drones have emerged as a promising solution for on-demand delivery, effectively alleviating urban traffic congestion. In this relay-based system, orders are fulfilled via a three-stage relay: a courier transports the parcel to a nearby station, a drone airlifts it to a proper station, and another courier finally delivers it to its destination. We formally define this scenario as the Dynamic Multi-Echelon Pickup and Delivery Problem (DM-PDP). Solving DM-PDP is challenging due to cross-stage dependency and stage-wise differentiation. Classical operations research methods face bottlenecks in computational complexity and time efficiency when dealing with large-scale dynamic problems. Moreover, existing reinforcement learning methods are typically tailored for single-echelon problems; this not only mandates stage decoupling that prevents joint optimization, but also fails to address the differentiation across stages. To address these issues, we propose MoE-Pointer, a unified reinforcement learning framework that reformulates DM-PDP into a sequence-to-sequence generation task. Specifically, we employ a Transformer Encoder to extract entity states and a Pointer-based Transformer Decoder to auto-regressively generate joint action sequences, thereby capturing cross-stage dependencies for end-to-end joint optimization. To address stage-wise differentiation, we introduce a Spatio-Semantic Relational Attention module to encode heterogeneous interactions and integrate a Mixture-of-Experts (MoE) architecture into the decoder to learn stage-specific policies. Moreover, we introduce a Prior-Guided Soft Mask to guide exploration within the exponentially large action space. Experiments demonstrate that MoE-Pointer outperforms existing baselines in solution quality, while significantly reducing computation time compared to operations research methods. Our code is available at https://github.com/luiluizi/MoE-Pointer.
