On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
It is shown that this gating mechanism causes SSMs to first learn an in-weights"memorization"solution, while delaying, or even preventing, convergence to a correct in-context learning solution, and that gating is often beneficial for improving generalization to long sequence lengths.
W. Tong, Aryo Lotfi, Emmanuel Abbé et al.
· 0 citations