MigMoE: Task-Aware Expert Migration for Faster and More Balanced Expert-Parallel MoE Inference

Abstract

Sparse Mixture-of-Experts (MoE) architectures scale LLM capacity, but serving them with expert parallelism often suffers from the straggler effect caused by skewed token routing and uneven placement of hot experts across GPUs. Existing methods mitigate stragglers by adjusting token-to-expert distributions or by replicating hot experts across GPUs, but they either trade output quality for lower tail latency or rely on an expert placement tuned to a specific routing profile. However, real-world LLM serving is multi-task and time-varying, so a placement that works well for one profile can become inefficient as the workload shifts. Motivated by the observation that routing patterns are task-specific, we propose MigMoE, a task-aware expert migration framework that dynamically adjusts expert placement for multi-task expert parallel MoE inference to balance loads across GPUs. In the offline phase, MigMoE profiles each task type and generates a migration plan under a migration budget using our Dynamic Programming-based planning algorithm. In the online phase, MigMoE matches incoming requests to the most suitable task profile and switches to the corresponding expert placement by executing the planned migrations. Experiments on three MoE LLMs show that MigMoE consistently outperforms two state-of-the-art baselines, EPLB and HarMoEny, improving throughput by up to 56.4% and 68.4%, respectively, while alleviating stragglers and maintaining comparable answer quality.