POSTER: Prediction-Enhanced Expert Prefetching and Eviction for MoE Offloading via PRED-MoE
TL;DR
PRED-MoE uses a lightweight predictor to assess which experts are most likely to be required for prefetching and ranks experts by their near-future activation likelihood for eviction, and indicates that it reduces the TPOT of vLLM and HuggingFace by up to 3.02× and 28.62× respectively.
Abstract
Mixture-of-Experts (MoE) models improve scaling by activating a small number of experts per token. However, the combined memory requirements of all experts may exceed the GPU's available high-bandwidth memory (HBM) during inference. Inference frameworks such as vLLM and HuggingFace address the problem by offloading experts to CPU memory and moving them to the GPU's HBM as needed. While enabling inference of large models with limited HBM, this CPU-GPU traffic overhead slows down token generation. We present PRED-MoE, a novel prefetching and eviction framework for MoE offloading. PRED-MoE uses a lightweight predictor to assess which experts are most likely to be required for prefetching and ranks experts by their near-future activation likelihood for eviction. Preliminary evaluation over the Qwen3-30B-A3B model and the MMLU-Pro dataset indicates that PRED-MoE reduces the TPOT of vLLM and HuggingFace by up to 3.02× and 28.62× respectively.