RETRIEVAL BEATS FINE-TUNING: ADAPTING A KAZAKH SMALL LANGUAGE MODEL FOR HISTORICAL QUESTION ANSWERING

Abstract

Large language models (LLMs) have shown a strong ability to solve a wide range of language tasks. However, they became less practical for subject-specific applications, especially when dealing with low-resource languages. This paper shows an example of adaptation of small language models (SLMs) to solve subject-specific language tasks in the history field. As a base model, we used the small vision language model (VLM) Qolda, developed by the Institute of Smart Systems and Artificial Intelligence (ISSAI) at Nazarbayev University. In the paper, 4 architectures are compared: 1) base model Qolda; 2) Qolda model with supervised fine-tuning (SFT) created using 48 000 instruction pairs retrieved from Wikipedia; 3) Qolda with retrieval-augmented generation (RAG) framework based on 29 digital history textbooks with 2.25 million indexed tokens; 4) Qolda combined with both SFT and RAG. To establish whether the research findings follow the same patterns across other Kazakh language models, we apply our evaluations to two external larger baseline models: 1) KazLLM and 2) Sherkala-Chat. All architectures are evaluated on 1,081 multiple-choice questions (MCQs) from the Unified National Test (UNT) in the history of Kazakhstan and on a 500-query open-ended test set. Results show that Qolda+RAG achieves 74.93 ± 0.43% MCQ accuracy and a mean quality score of 3.34 ± 1.64. In these settings, SFT acts as a response formatter, reducing the average response length from 51.8 to 14.64 words. Qolda-SFT+RAG completes inference in roughly half the time of Qolda+RAG with comparable quality of 3.10 ± 1.42. The study findings provide a foundation for the design and development of future Kazakh-language-supported AI agents across broad application domains. The project implementation, datasets, and evaluation methodology are available on the project’s GitHub page with the corresponding Hugging Face datasets: https://github.com/IS2AI/history-qolda-ui.