A Multi-Branch Transformer Based CrossAttention Framework for Computer-GeneratedImage Detection

Abstract

The rapid development in deep learning-based generative softwares and image rendering tools has led togeneration of massive photorealistic digital content – Fake images, fake videos, fake speech, etc. Such fake digitalmedia may result in communication of misinformation, forgery of digital data, and losing trustworthiness in theinformation source. This poses a significant challenge to the field of digital forensics’ techniques, to which our presentwork attempts to make a contribution, by addressing the problem of differentiating AI-generated images from realphotographs, using transfer learning and multi-branch fusion model. We propose a multi-branch model that integratestwo pre-trained Vision Transformer models (DINO (self-distillation with no labels) and Contrastive Language – ImagePretraining (CLIP)) to extract complementary global features, along with a forensic and a hand-crafted feature branch,which extract low-level discriminating cues. These features are complimentary to each other and hence contribute inimproving the robustness and performance of the model. These features from the four branches are adaptively weightedand combined by a cross-attention module, to give a fused and rich embedding. The model is further optimized by usingaugmentation-invariant loss, center loss and supervised contrastive loss in addition to the cross-entropy loss function.This framework achieves improved accuracy of 95.83% on PRCG dataset, 96.79% on CIFAKE dataset and 99.15% onGenImage dataset as compared to baselines. It also achieved stable cross-generator performance and enhancedrobustness against real world corruptions like Blur, Noise, Compression, and others. The experimental results show agood separability between the classes, and enhanced performance on publicly available datasets.