Cross-Modal Inconsistency Mining for Detecting and Grounding Multi-Modal Media Manipulation

Abstract

Generative models can now forge the visual and the textual channel of a news item at the same time, producing multi-modal fake media that erodes public trust and threatens the integrity of the information ecosystem. A trustworthy defense must go beyond a binary real or fake verdict and also ground the manipulation, that is, localize the tampered image region and the tampered text tokens. We observe that different manipulations leave complementary traces: face swap and face attribute edits leave low level forensic artifacts on the image, whereas swapped or rewritten text produces a high level semantic inconsistency between the caption and the visual content. Building on a pretrained vision language dual encoder that we fine tune end to end, we propose Cross-Modal Inconsistency Mining (CMIM), a lightweight framework that extracts image forgery cues through a noise residual forensic stream, mines image text inconsistency as the residual left after text tokens attend to image patches, and reuses these signals to detect and ground manipulations jointly. On the large scale DGM4 benchmark, a controlled ablation shows that inconsistency mining, not merely added capacity, is the principal source of the improvement, giving 1.2 point higher detection AUC and 3.2 point higher fine grained mAP over a strong matched baseline. With a CLIP-ViT-L/14 backbone the unified model reaches 0.95 detection AUC and provides image grounding at 0.71 IoU together with text token grounding at 0.54 F1, capabilities that a detection only baseline structurally cannot offer. By making automated verification of digital media more accurate and interpretable, this work contributes to United Nations Sustainable Development Goal 16, Peace, Justice and Strong Institutions.