Explainable Multi-Cue Attention Networks for Deepfake Face Detection and Human-Aligned Decision Support
Abstract
Generative face models are becoming harder to distinguish from real photos, as the face images produced by adversarial and style-based models become more realistic. To assess the efficacy of StyleGAN2 for generating human faces, a recent large-scale study of human subjects was conducted, where unaided observers correctly labelled StyleGAN2 face images with a success rate of just 62.18%, and there was no correlation between accuracy and confidence, and barely significant improvement in accuracy when simple interventions based on style advice were performed. Directly inspired by these results, this paper proposes the Explainable Multi-Cue Attention Network (EMCAN), which (i) combines spatial, frequency-domain and landmark-symmetry cues using an attention mechanism learned from data, (ii) explicitly incorporates a confidence-calibration term ensuring that the reported level of confidence reflects the accuracy of the prediction, and (iii) exposes per-region saliency maps aligned with the 10 human-interpretable "tell-tale" cues catalogued to serve as an overlay for human reviewers to assist in their decision-making process instead of a black-box classifier. We explain the architecture, loss formulation and an evaluation protocol that is based on the same FFHQ/StyleGAN2 image pool. To demonstrate expected behavior and to inspire an empirical evaluation that's pre-registered, we report results from a preliminary simulation study calibrated to the human accuracy, confidence and reliability per cue statistics reported. Simulated results show that the proposed model has a significant deviation from humans when humans are merely advising them statically, and suggest that the cuealigned saliency map might reduce the drift of the false positives.