Optimizing Few-Shot Learning in Pruned Large Language Models with Task-Specific Prompts

Abstract

Few-shot learning enables large language models to efficiently perform tasks given only a limited number of labeled examples. However, training these models entirely from scratch requires substantial computational resources, making it challenging for many organizations to fully leverage their potential. This thesis explores how structured pruning, task-specific prompting, and parameter-efficient fine-tuning can be combined to preserve few-shot learning capabilities in compressed LLMs, while also extending their utility to real-world recommendation systems. In this research, we propose the Tailored LLM framework, which first reduces model size through structured pruning and then enhances few-shot learning performance using carefully designed prompts. We experiment with pruned variants derived from two original LLaMA models of 7B and 13B parameters. Specifically, from the 7B parameter model, we obtain pruned variants with 5B, 4B, and 2B parameters, and from the 13B parameter model, variants of 11B, 9B and 7B parameters. Fine-tuning is conducted on task-specific datasets with a small subset of weights modified using the Low-Rank Adaptation (LoRA) method. One of the important contributions of our work is the careful crafting of prompts tailored explicitly for each few-shot learning task, significantly improving task accuracy after pruning. Experimental results demonstrate the effectiveness of our method. Fine-tuning a 7B LLaMA model pruned by 50% for less than one hour achieves a classification accuracy of 95.68% at a 20% compression ratio, and 86.54% at a 50% compression ratio, utilizing only 50 training examples. Further evaluation reveals that, even after aggressive pruning, the models retain more than 65% of the baseline accuracy across the classification tasks. Additionally, we demonstrate the effectiveness of LLM-derived embeddings in a CTR prediction pipeline. By combining contextual embeddings from a fine-tuned LLaMA model with collaborative signals from a separately trained Deep Interest Network (DIN) model, we achieve strong performance across multiple recommendation datasets. This hybrid setup highlights the versatility of compressed LLMs in both language understanding and real-world ranking tasks. This thesis confirms that optimized task-specific prompting, combined with structured pruning and targeted fine-tuning, considerably improves few-shot learning performance. This approach provides an efficient pathway for adapting large language models to specific tasks, particularly valuable in scenarios where computational resources are limited.