Training datasets tailored for domain-specific large language models: challenges, methods and trends

Abstract

Adapting LLMs to specialized domains critically depends on high-quality training data, making dataset construction a decisive factor in domain performance. This survey reviews methods for constructing domain-adaptation training datasets through a structured, data-centric analysis. Based on a comparative analysis of representative domain-specific LLMs, we organize the literature around three technical dimensions that address limited data availability, heterogeneous quality and inefficient utilization: data synthesis, data selection and data composition. For data synthesis, we examine paradigms based on instruction rewriting, unlabeled-corpus reconstruction, retrieval augmentation and knowledge-graph-guided generation. For data selection, we trace the progression from heuristic rule-based filtering to model-aware methods that estimate sample quality and training utility. Data composition encompasses mechanisms that control the effective training-data distribution by determining which data are used, how strongly different data contribute to optimization and when each data type is introduced. We further examine catastrophic forgetting during domain adaptation and summarize data-centric mitigation strategies based on replay, mixture design, data ordering and stage-wise allocation. Overall, this survey provides a systematic reference for constructing domain-adaptation training datasets and supports the transition of LLMs from general-purpose foundation models to domain-specialized systems.