Reasoning-BG: A Hybrid Benchmark for Long-Text Understanding and Reasoning in Bulgarian

Abstract

The paper introduces the Bulgarian benchmark Reasoning-BG, designed to evaluate how accurately large language models (LLMs) solve Bulgarian long-text comprehension tasks involving different comprehension demands. It was developed using a hybrid methodology that combines automated generation of four-option multiple-choice questions by the Gemma-3-27B-IT model, applied to 232 Bulgarian texts, followed by validation, correct-answer determination, and editing by human experts. The question design, which addresses relatively long texts, is inspired by the Programme for International Student Assessment (PISA) framework, which targets reading-comprehension tasks such as retrieving information, interpreting and integrating ideas, and evaluating and reflecting on content. The benchmark provides a framework for evaluating model-driven, cognitive-like processes, shifting the focus from factual recall to the analysis of complex interpretative and logical performance in Bulgarian.