Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs
The experimental results show that even 130M-parameter models benefit from including the MTP task in the pre-training objective, and hold even under severe data constraints, as demonstrated on both zero-shot benchmarks and downstream tasks.
Ansar Aynetdinov, Alan Akbik
· 1 citation