Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

Fine-Tuning of Transformer models with Frames

The experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.

Parameter-Efficient Fine-Tuning of Transformer Models: An Empirical Study of LoRA Optimization for High-Cardinality Intent Classification

Although parameter-efficient fine-tuning significantly reduces the computing cost of deep models, default configurations are insufficient to perform as good as full fine-tuning for challenging large-cardinality intent detection problems with 77-151 intents. Thus, this work presents the S1 configuration that is proposed to remedy such performance degradation by defining the state-of-the-art low-rank adaptation. Instead of being constrained by the conventional formulation, this approach uses the minimal possible rank-8 adapter, full linear module coverage, and a learned learning rate. Extensive ablations offer two important discoveries that structural module coverage has more impact than mere adapter rank, and high learning rate is indispensable to provide enough convergence with the limited number of parameters. We show in the experiment that this configuration manages to restore the model performance to the 93.73% and 90.18% on Banking77 and CLINC150 respectively. S1 configuration attains the baseline parity, while only updating 1.56%-1.60% total trainable parameters with a maximal 2.96GB memory. This proves that it is possible to train a high-accuracy transformer on the hardware targeted at consumers, for example the NVIDIA RTX 5060 Ti.

Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

An Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained LLMs into depth-sparse models without full retraining and consistently yields smaller accuracy degradation than strong baselines including MoD, D-LLM, and DLO.

Fisher-Based Layer-Wise Adaptive Sparsification for Efficient Pruning of Large Language Models

Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

The results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.

Low-Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Language Models

Modern natural language systems rely on large language models, whose sheer size makes full fine-tuning costly in computation, graphics processing unit (GPU) memory, and storage. Low-rank adaptation (LoRA) sidesteps most of that cost. It keeps the pre-trained weights frozen and captures each task-specific change as the product of two smaller matrices, so adapting a model reduces to a low-rank decomposition. This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them. It first explains why the low intrinsic dimension of fine-tuning makes low-rank updates effective, then compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA). Across published studies, these methods come close to full fine-tuning accuracy while updating well under one percent of a model's parameters in some settings. For reference, LoRA cuts the trainable parameter count of Generative Pre-trained Transformer 3 (GPT-3) by four orders of magnitude, and QLoRA brings a 65-billion-parameter model within the memory of one 48 GB card. Open problems remain in choosing the rank, comparing results across studies, limiting quantization loss, and combining multiple adapters without interference. Ultimately, an established piece of linear algebra, approximating high-dimensional objects in low-dimensional subspaces, is what keeps the adaptation of very large models affordable.