Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRA
This work rigorously establishes trainability of Low Rank Adaptation models under stochastic methods, and proves that for a class of mild regularizations, the empirical regression loss on a attention layer and LoRA on a shallow neural net, both induce Poincar's inequality for the corresponding Gibbs'asure.
Zhengkai Sun, Dibyakanti Kumar, A. Frangi et al.
· 1 citation