Ken-Tye Yong

Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space

Optimizer momentum is usually stored as a parameter-sized moving average of past gradients, which makes history costly and fixes each past signal in the coordinates in which it was computed. We introduce Backpropagated Output Momentum (BOM), which instead stores a compact moving average of prediction errors at the mode...