HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion
HERO-MoE, Historical Expert ROuting with Scale-Preserving Fusion, a routing framework that injects historical routing priors into MoE routers by reusing detached, dense routing distributions collected from preceding MoE layers by reusing detached, dense routing distributions collected from preceding MoE layers.