GMTW-Ro: a deterministic benchmark for evaluating large language models on grounded Romanian tasks
TL;DR
GMTW-Ro is introduced, a benchmark designed to evaluate whether large language models can reliably follow complex instructions in Romanian, rather than merely produce fluent text, and raises important questions about how current language adaptation pipelines preserve instruction-following and structured reasoning capabilities.
Abstract
We introduce Grounded Multilingual Task Worlds for Romanian (GMTW-Ro), a benchmark designed to evaluate whether large language models can reliably follow complex instructions in Romanian, rather than merely produce fluent text. Existing Romanian benchmarks largely rely on multiple-choice formats, answer extraction, or model-based evaluation, which struggle to assess multi-constraint reasoning and structured task completion. GMTW-Ro addresses these limitations through grounded task worlds: fully specified environments in which model outputs are verified via deterministic, programmatic checks. The benchmark spans four task domains—travel planning, calendar scheduling, context-grounded question answering, and dietary menu planning—requiring both a structured JSON plan and a natural-language explanation in Romanian. Evaluation is decomposed into three orthogonal metrics: Understanding (U), measuring constraint adherence and instruction-following; Generation (G), assessing Romanian text quality through diacritic accuracy, language purity, and code-switching absence; and Faithfulness (F), quantifying consistency between generated plans and their explanations. All instances are automatically verified as solvable using backtracking algorithms. We release two curated datasets: a standard benchmark of 500 instances and an adversarial set of 300 instances with heightened constraint complexity, alongside the complete evaluation toolkit and a purpose-built Romanian NLP library. Evaluation of 11 models reveals substantial performance variation (58.6%–90.7%) and exposes a pronounced knowledge–behavior gap, where models with fluent Romanian generation nevertheless fail core reasoning tasks. Most notably, Romanian-finetuned models underperform their base counterparts: RoLlama3.1-8B scores 20.1 percentage points below Llama-3.1-8B, with structured JSON output success dropping from 95 to 44%. These results raise important questions about how current language adaptation pipelines preserve instruction-following and structured reasoning capabilities.