SS-AV-InDF: a self-supervised multimodal framework with local and global feature modeling for deepfake detection in Indian media

Abstract

Information integrity is seriously threatened by the growing number of deepfakes, but current detection techniques frequently have weak generalization and demographic bias due to the biasness of datasets. This research presents a two-stage self-supervised multimodal architecture that learns inherent audio–visual correspondence especially for Indian media. The proposed approach learns multimodal features both locally and globally using cross attention along with transformer-based CLIP and SimCLR modals. This contrastive learning based SSL technique uses InfoNCE loss to learn the representations during pretext task. This study compares the Transformer-based (CLIP/SimCLR) SSL designs with various SSL techniques, such as a ResNet-50 baseline, an Xception-based model, and custom handcrafted features with traditional classifier. Further, a comparative architecture analysis of the proposed SSL i.e. Transformer-based (CLIP/SimCLR) architectures with other methods for SSL has been conducted. To assess the resilience of the model for the Indian dataset, all SSL techniques are tested on the varied InDeepFake dataset. The suggested approach (CLIP+SimCLR, local pipeline) outperforms all architectures., based on corrected leakage-free performance, subject-disjoint stability, cross-dataset generalization, demographic fairness, and training efficiency. Our suggested framework outperforms ResNet-50, Xception, and handcrafted-feature baselines with significantly smaller demographic performance disparities, achieving 97.92% accuracy and 99.95% AUC on InDeepFake under a strict subject-disjoint evaluation protocol with zero identity overlap between training and test sets. These findings suggest that localized artifacts are the best way to characterize contemporary deepfakes, with patch-based detection addressing intrinsic instability for real-world implementation. Further to test the generalizability of the proposed SSL architecture, we used three datasets containing Indian languages and video i.e. InDeepFake, FakeAVCeleb and HAV-DF.