MACHINE LEARNING-BASED DETECTION OF AI-GENERATED CONTENT IN STUDENTS’ WRITTEN WORKS
TL;DR
The results support the use of stylometric indicators as an interpretable supplementary signal for educational text analysis, while also emphasizing that detector outputs should not be treated as definitive proof of AI use in individual student work.
Abstract
The rapid spread of generative artificial intelligence has complicated the interpretation of authorship in educational writing and increased the need for transparent methods that distinguish machine-generated from human-written text without treating automated detection as conclusive evidence of misconduct. This study compares lexical and content-independent stylometric approaches to detecting AI-generated educational essays. The experiments used 44,868 English-language texts from the DAIGT V2 multi-source corpus, including 27,371 human-written and 17,497 AI-generated essays. The lexical representation used TF-IDF unigrams and bigrams with a Linear Support Vector Machine, while the content-independent representation consisted of fourteen structural and stylometric features classified with Logistic Regression. Human and AI texts were additionally compared using the Mann–Whitney U test and effect-size estimates. The TF-IDF model achieved 99.69% accuracy, 99.60% F1-score and 99.98% ROC-AUC on a stratified held-out test set. The stylometry-only model achieved 87.90 ± 0.40% accuracy, 84.34 ± 0.48% F1-score and 95.08 ± 0.17% ROC-AUC under five-fold stratified cross-validation. Average word length showed the strongest selected stylometric separation, while differences were also observed in text length, lexical diversity and punctuation use. The findings show that lexical information provides highly discriminative signals in the analyzed corpus, but substantial predictive information remains even when direct vocabulary is removed. The results support the use of stylometric indicators as an interpretable supplementary signal for educational text analysis, while also emphasizing that detector outputs should not be treated as definitive proof of AI use in individual student work. Robustness checks showed 99.75% accuracy after near-duplicate filtering, 96.46% accuracy for unseen prompts, and 95.39% accuracy when Llama-70B was held out, indicating measurable distribution sensitivity. The contribution is therefore a robustness-oriented comparison of established lexical and interpretable stylometric representations rather than a new classifier.