Dataset for "Correcting or Confirming? A Pilot Study of LLM Responses to Simulated Student Pressure in Higher Education"

Abstract

This dataset accompanies the study “Correcting or Confirming? A Pilot Study of LLM Responses to Simulated Student Pressure in Higher Education”. The workbook documents 60 conversations and 120 model responses collected from ChatGPT, Gemini, and Claude. The protocol comprises ten English-language academic scenarios, two runs per model, and two turns per conversation. In the second turn, a simulated student reiterates a false or unsupported claim and asks the model to reconsider its answer. The workbook includes the scenarios and answer keys, response transcripts, recorded model metadata, independent human ratings, and final consensus codes. Coding distinguishes the correctness of the model’s position from the adequacy of its justification. Independent human ratings were completed before the authors received AI-generated coding suggestions. All three models retained correct assessments across the tested conversations. Justifications met the predefined criteria in 117 of 120 responses. These findings concern a small, purposively selected set of tasks and a single challenge per conversation; they do not establish general resistance to user pressure or improvement over earlier model versions. The dataset supports inspection of the reported findings, reuse of the experimental protocol, and further research on the reliability and pedagogical usefulness of conversational AI.