A Dataset for Multidimensional Evaluation of French Synthetic Texts Generated by SLMs

Abstract

This dataset comprises synthetic news descriptions in French, generated from real newspaper headlines using a controlled Small Language Models (SLM) pipeline. The generation process follows three distinct configurations: Retrieval-Augmented Generation (RAG) and generation without external context (NO RAG) and QLoRA-based fine-tuning (FT). In the zero-shot configuration, the base model generates each description using only the corresponding headline and generation instruction. In the RAG configuration, the same base model additionally receives contextual information retrieved from a historical French news corpus. In the fine-tuning configuration, the models are adapted using headline-description pairs from the historical corpus through QLoRA and subsequently used to generate descriptions without external retrieval. For each configuration, three temperature settings (0.9, 0.6, and 0.3) were applied to control the variability and determinism of the generated outputs. The generation process relied on a corpus of real-world French news articles collected between February 16 and March 26, 2026. To prevent information leakage, the source data was chronologically strictly partitioned: Temporal Window Timeframe News Items Description Knowledge Window 16 Feb 2026 – 18 Mar 2026 7,767 Served as the RAG knowledge base and the QLORA fine-tuning dataset. Generation Window 18 Mar 2026 – 26 Mar 2026 2,071 Reserved exclusively for synthetic text generation and evaluation. The experimental pipeline originally generated 55,917 synthetic descriptions across 27 configurations, derived from three Small Language Models (Ministral-3B, TinyLlama-1.1B, and Qwen2.5-0.5B), three generation strategies, and three temperature values. To ensure copyright compliance, exact verbatim reproductions of the input headlines were removed, resulting in a final publicly released dataset of 54,751 synthetic descriptions. The dataset is distributed in the fr_slm_multieval.zip. Inside the archive, the data is organized into newline-delimited JSON (.ndjson) files. Each file corresponds to a specific experimental configuration, allowing researchers to isolate model behaviors. The files follow the naming convention: [strategy]_[RAG-status]-[model]-[temperature].ndjson where [strategy] specifies the model adaptation state: zs: base model, used either in zero-shot or RAG generation. ft: QLoRA fine-tuned model. Examples of the configuration files included in the file: ● ft_NO-RAG-ministral-0.3.ndjson ● zs_RAG-qwen-0.9.ndjson