EVAL Study Dataset and Supporting Materials: Clinical Safety of Large Language Model Responses to Simulated Iraqi Arabic Breast Cancer Scenarios and Agreement Between GPT-5.1 and Human Experts

Abstract

This repository contains the datasets and supporting methodological materials for a multi-phase evaluation of the clinical safety and performance of large language model (LLM) responses to simulated Iraqi Arabic breast cancer scenarios and agreement between GPT-5.1 automated evaluations and human experts. The study comprised 50 simulated, single-turn Iraqi Arabic breast cancer scenarios across four clinical categories: oncologic emergencies (n=10), chemotherapy-related adverse effects (n=20), medication/herb/supplement interactions (n=10), and lifestyle/supportive care (n=10). Fourteen LLM configurations generated one response per scenario, yielding 700 model responses. Each response was assessed using seven domain-specific GPT-5.1 evaluator configurations, producing 4,900 automated evaluation outputs. Human validation included 80 responses generated by four selected models across 20 scenarios chosen by stratified random sampling. Two blinded clinical experts independently rated all responses. Responses with a ≥2-point disagreement in any graded domain or any binary flag disagreement underwent third-expert adjudication; 39/80 responses were adjudicated. The resulting ratings constitute the final human reference. Repository materials include the simulated scenarios and structured reference standards, all 700 model-generated responses, automated GPT-5.1 evaluation outputs, human expert ratings and adjudication records, the verbatim response-generation and evaluator prompts, the study scoring framework and worked example, human-evaluator instructions, model characteristics, computational-resource information, and a data dictionary. No real patient conversations or identifiable patient data are included. Analysis code is not included in this repository.