DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving
Serving a Mixture-of-Experts (MoE) model pays for expert precision twice: the resident footprint pays for every expert the router can reach, while each batch-one decode step pays only for the experts it selects. These are two different rate functionals over the same per-expert widths, yet published per-expert MoE alloc...
Meng-Qi Guo, Zhe Wei, Yuan Yuan et al.
· 0 citations