Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency. Deviating from the widely use...
Dian Jiao, Jiaxin Duan, Shuai Zhao et al.
· 0 citations