Rethinking Contextualization by Reinterpreting Attention Head Channels
It is found and estimated that different words carry different amounts of information, and less-informative words tend to absorb more contextual information, and finer-grained selectivity enables more precise routing to promote information transmission between matched words.
Hakaze Cho, Hao-Lin Yang, Zhun Sun et al.
· 0 citations