What Transfers and What Collapses: A Cross-Generator Study of AI-Generated Image Detection with Corrected Evaluation

Abstract

Version 3 (10 September 2026). Corrected a benchmark-size error: the paper stated 14 generators throughout while the actual assembled and evaluated benchmark (and the enumerated list) contains 13; all occurrences are corrected to 13. Also corrected the Table 4 cross-architecture claim: not 'every detector' scores higher on ProGAN than on unseen diffusion generators -- the frozen-CLIP probe is an exception (0.892 on ProGAN vs. 0.917 on unseen diffusion), now noted explicitly. Found by an independent adversarial audit; no other results, figures, or numbers changed. Version 2 (10 September 2026). Added an explicit Author Contribution and Disclosure section (new Section 8): the study design, protocol, and interpretation are the author's own; the implementation (detector code, evaluation harness, statistical tooling) and initial paper drafting were produced with AI coding assistance (Claude Code) under the author's direction and review. The author verified the reported numbers against the released code and is responsible for the correctness of all claims. No results or figures changed. A cross-generator study of AI-generated image detection on a 14-generator benchmark of modern systems (Stable Diffusion 1.3/1.4/2/XL, SD3, FLUX.1-dev/schnell, DALL-E 2/3, Midjourney v5, Imagen 3, GLIDE, Adobe Firefly), under a leak-free leave-generators-out protocol with corrected metrics and bootstrap confidence intervals. Findings. In-distribution accuracy does not predict cross-generator accuracy: a 2-D spectral detector scores 0.795 in-distribution but 0.523 (chance) on unseen generators, and a fine-tuned CNN drops to chance on its worst unseen generator; a hand-crafted physics detector goes confidently below chance (0.258, fingerprint inversion). Only frozen foundation features generalize (frozen-CLIP probe, 0.917), and a one-class real-manifold model structurally cannot sign-invert (worst-case floor 0.611). The collapse is governed by training-generator diversity (leave-one-out recovers the CNN to 0.876 and the CLIP probe to 0.960). A cross-architecture probe shows detectors do better on an older GAN than on unseen diffusion. Improvement. Feature-space extensions (DINOv2 ensembling, generator-direction removal, one-class fusion, reconstruction fusion) do not beat the frozen-CLIP probe, but simple 5-crop test-time aggregation does, by a paired-bootstrap-significant margin: 0.914 -> 0.949 (three-seen, 95% CI [0.028, 0.042]) and 0.961 -> 0.972 (leave-one-out), with the largest gains on the hardest generators. We also document an AUC-orientation evaluation bug that returns 1-AUC and silently inverts results. Code and benchmark: https://github.com/theFinex/cross-generator-ai-detection