Decoupling Token Roles in Autoregressive Pretraining
Using controlled corruption, this work decouple these two roles and finds a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context.
Su-Qin Yuan, Runqi Lin, Ke-Yu Lin et al.
· 0 citations