A Qualitative Evaluation of ChatGPT-Generated Academic Writing in Higher Education

Abstract

The purpose of this study is to evaluate academic texts generated by ChatGPT, in terms of academic language use, structural integrity, content quality, and source reliability. Within this scope, ChatGPT-4 was prompted to produce an academic text in Turkish on a specific topic, and the resulting text was assessed using a rubric developed by the researchers. Subsequently, the text was qualitatively analyzed through detailed document analysis. The findings reveal that while ChatGPT generally demonstrates adequate performance in academic language use (such as objective tone, academic style, and proper application of academic grammatical structures), certain shortcomings were observed in critical analysis and adherence to APA 7 formatting rules. Although the organizational structure of the text presents a largely coherent and logical flow, minor structural weaknesses (such as the lack of a roadmap in the introduction) were identified. In terms of content, arguments remained mostly superficial, the literature was not examined in depth, and some references could not be verified. Furthermore, instances of information being inconsistent with or misinterpreted from the cited sources were detected. These results indicate that generative AI tools can be supportive in academic writing, but their outputs require careful evaluation.