OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading
OLED-MoE is proposed, an expert offloading system that uses confidence-guided inter-iteration prediction to retain high-value experts in GPU memory without introducing extra prefetch traffic and compensates unavoidable cache misses through CPU-GPU cooperative execution.
Jing Xiao, Jia-Yue Wang, Yi-Tao Hu et al.
· 0 citations