RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
The RunningTensor is introduced, which generalizes this memory to an order-o tensor, updated by a rank-1 outer product and read by contracting against $o-1$ vector queries, and improves performance on language-understanding and non-synthetic retrieval tasks, suggesting that higher-order recurrent state can provide usef...
Luca Herranz-Celotti, Vincent Guigue
· 1 citation