Length Generalization Needs Proper Regularization
Length generalization is the ability of sequential models to perform well on context lengths unseen during training. In this work, we show that the challenge of achieving length generalization is related not only to architectural choices such as positional encoding and the attention mechanism but also to the training p...
Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins et al.
· 0 citations