Symbolic Preference Distillation: Advancing Small Language Models for Mental Health Analysis.

Abstract

Large language models (LLMs) have demonstrated strong performance in mental health analysis tasks when equipped with advanced reasoning capabilities. However, their substantial parameter sizes and high computational demands present significant barriers for routine clinical use. Recent studies have explored reasoning distillation as a means to transfer these capabilities to small language models (SLMs). However, SLMs often struggle with complex reasoning tasks due to their limited capacity to model both general cognitive abilities and specialized domain knowledge. In this paper, we propose symbolic preference distillation (SyPD), to enhance the complex reasoning abilities of SLMs in mental health analysis tasks. First, to handle challenging or ambiguous cases, we introduce a reasoning optimization strategy that leverages specialized domain knowledge to perform SLMs' error analysis and generate symbolic knowledge via a teacher. Second, to further boost SLM's reasoning ability, we propose a preference distillation method that guides an SLM to align with high-quality and clinically relevant reasoning derived from the teacher LLM through preference signals and symbolic knowledge, without requiring access to the teacher's output probabilities. By anchoring the optimization to the model's own pre aligned distribution, our method enables post-hoc correction of failure cases while gaining domain-specific knowledge. Experimental results demonstrate that our proposed SyPD, with only 1.1 billion parameters, achieves an average weighted F1-score of 0.815 on mental disorder diagnosis on the interpretable mental health instruction (IMHI) bench mark. It outperforms the state-of-the-art instruction-tuned MentaLLaMA-Chat-13B model by 6.14%, and the few-shot tuned GPT-4 model by 15.44%.