Detecting AI-Generated Text and Code: An Empirical Study of Cross-Generator and Cross-Domain Generalization

Abstract

Large language models (LLMs) now generate fluent natural language and source code, creating challenges for authorship attribution, academic integrity, and software supply-chain security. Most existing detectors for AI-generated content are evaluated separately on natural language or source code, often under matched train–test conditions that can overestimate real-world reliability. We present a paired-prompt benchmark for human-versus-machine detection across English text, Python code, and mixed text–code documents. The benchmark includes 22,141 instances from HC3, CodeSearchNet, MBPP, and HumanEval across training, validation, and test partitions, plus Mix-Eval, a mixed-content set of 997 Jupyter-notebook-style samples. We evaluate RoBERTa-large for text, GraphCodeBERT and CodeBERT-base for code, a unified RoBERTa-base detector trained on both modalities, and zero-shot baselines. Fine-tuned detectors achieve near-perfect in-distribution performance, with AUROC 1.0000±0.0000 and accuracy above 99.5%. Across five instruction-tuned generator families of varying size (3.8B–7B) and architecture, with the human and problem distributions held fixed, cross-generator transfer causes negligible degradation (AUROC spread 0.0002; drops of at most 0.0003). In contrast, domain shift is the main failure mode: on MBPP+HumanEval, GraphCodeBERT drops to 0.85±0.02 AUROC and CodeBERT-base to 0.67±0.02. On Mix-Eval, the unified detector outperforms a routed text–code pipeline by 21 AUROC points (0.96 vs. 0.75), largely because of router failures on mixed inputs. Training-time augmentation improves low-false-positive performance, while legacy supervised detectors show systematic class inversion on modern LLM outputs. These results show that reliable deployment requires cross-domain evaluation, mixed-content testing, and calibration beyond in-distribution accuracy.