SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving
SlimWise is presented, a serving framework that tailors the expert pool to each inference phase that improves decode throughput by up to 1.81x at 50% expert pruning with minimal accuracy loss and shows that benchmark accuracy can conceal substantial pruning-induced changes in generation length.
Gunho Park, Kyoungho Jeun, Juntaek Oh et al.
· 0 citations