LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes
LRMoE is presented, a memory-aware MoE training system that jointly recalibrates backward demand and balances the retained workload, and reduces cumulative MoE-layer training time to fixed-loss targets by 21.2-44.8% relative to SYMI.
Jia-Ming Hu, Zhi-Da Jiang, Z. Xing et al.
· 0 citations