LYRA: Low-Frequency Rank Adaptation via Factored DCT Coefficients for Parameter Efficient Fine Tuning of Transformers

Abstract

As pretrained language models grow, full parameter updates become prohibitively expensive, motivating parameter-efficient fine-tuning (PEFT). LoRA decomposes weight updates into low-rank factors, but its per-module cost $r(m{+}n)$ imposes a structural floor at rank one. FourierFT breaks this floor via frequency-domain coefficients, yet requires a dense inverse transform at each forward step. We propose LYRA (Low-frequencY Rank Adaptation), which parameterizes each weight update with a small set of low-frequency 2D discrete cosine transform coefficients, chosen separately along each axis and contiguous in its canonical form. This separable structure admits a factored forward pass of three small matrix multiplications, avoiding dense reconstruction entirely. On GLUE and SuperGLUE with the BERT-base and RoBERTa-base encoders, LYRA matches FourierFT at identical parameter and optimizer-memory budgets while achieving the lowest peak GPU memory among all methods tested.