Dohyeon Kim

Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts

Layer-wise Distribution Alignment (LDA), a lightweight inference-time correction that uses layer-wise calibration statistics to align reduced-routing representations with the default configuration, is proposed, recovering much of the performance lost induced by the distributional shift under reduced routing while prese...