Toward scalable generative AI: efficient language model distillation via zero-shot rationales

Abstract

This paper investigates an efficient approach for distilling Large Language Models (LLMs) into smaller, application-specific models using zero-shot Chain of Thought (CoT) rationale generation and Optimization by Prompting (OPRO). To address the challenges of deploying computationally intensive generative AI for narrow tasks or resource-constrained environments, the approach leverages LLM reasoning capabilities to generate both labels and natural language explanations for unlabeled data. By reducing reliance on human-generated annotations, the approach substantially lowers annotation requirements and prompting costs while maintaining comparable performance in the evaluated settings. We formulate distillation as a multi-task learning problem in which student models are trained to jointly predict labels and learn from teacher-generated rationales, with the goal of improving data efficiency and generalization. Building on established zero-shot Chain of Thought (CoT) prompting and the OPRO prompt optimization technique, we use teacher-generated rationales to reduce annotation token requirements and examine the associated performance and efficiency gains. Additionally, we systematically investigate how explanation properties affect distillation efficiency. Across natural language inference and question answering benchmarks, results indicate that near-optimal performance can be achieved even when rationales are provided for only a subset of the training data, and that shorter explanations are often sufficient. These findings provide practical insights into the trade-offs between rationale generation cost and student model performance. Overall, this work contributes empirical evidence on the effectiveness and cost characteristics of rationale-based distillation for training compact, task-specific language models with minimal human intervention.