Massive Activation Gating Channel in Large Language Models
This paper finds that the emergence of massive activations is controlled by a single channel in the input embedding to a spike feed-forward network (FFN), and names this channel the massive activation gating channel (MAGC).
Min-Jia Mao, Shi Chen, Bo-Wen Yin et al.
· 0 citations