#machine learning
Feb 2025
You Do Not Fully Utilize Transformer's Representation Capacity
Layer-Integrated Memory (LIMe) is introduced, a lightweight extension that leverages existing key-value buffers and learns per-head, per-layer routing weights to integrate representations from previous layers to improve perplexity per FLOP and yield strong gains on synthetic tasks while preserving higher value-vector entropy and token separability.
Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky et al.
· arXiv.org · 4 citations