Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
OptEP is developed, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of...
Jelena Markovic-Voronov, P. Prastakos, Kayhan Behdin et al.
· 0 citations