Kai Wang

Dive Into the Implicit Biases of Low-rank Vision-language Alignment

This work systematically characterize the implicit biases introduced by low-rank adaptation during alignment and establishes two theorems showing that low-rank alignment induces preferences for parameter subspaces with flat gradients and feature subspaces robust to perturbations, providing a principled explanation for the observed structure-preserving behavior.