Justin Shi

BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferring model weights from accelerator high-bandwidth memory into on-chip SRAM. Mixture-of-experts (MoE) models reduce computation by activating only a small subset of experts...