David Ngwe Pouth

Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA

Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of fr...