PD-CoTD:A Prefix-Description Enhanced CoT Distillation Framework for Improving Mathematical Reasoning in Small Language Models
TL;DR
Experimental results show that PD-CoTD significantly enhances the mathematical reasoning abilities of SLMs and provides an effective solution for deploying powerful reasoning in resource-constrained environments.
Abstract
Large Language Models (LLMs) have strong capabilities in mathematical and logical reasoning. However, deploying LLMs is difficult for users with limited hardware. One solution is distilling reasoning skills from LLMs to Small Language Models (SLMs, ≤1B parameters). Still, student models often make five common errors: calculation errors, missing reasoning steps, symbol mistakes, semantic misunderstandings, and coherence errors.This paper proposes Prefix-Description-CoTD (PD-CoTD), an improved distillation method to address these issues. First, structured prompt templates guide LLMs to generate complete reasoning steps. This creates high-quality datasets to help SLMs learn effective reasoning patterns. Second, we introduce a step-by-step optimization loss function. This loss function separates reasoning path modeling from final answer prediction. It helps the model focus on reasoning clearly while optimizing results. Experimental results show that PD-CoTD significantly enhances the mathematical reasoning abilities of SLMs. It achieves state-of-the-art performance and greatly reduces the five common errors. PD-CoTD provides an effective solution for deploying powerful reasoning in resource-constrained environments. It also supports the broader application of lightweight models in academic research and education. Our code implementation and data are available at https://github.com/gameking2159/PD-CoTD .