A Two-Stage CNN-LSTM Framework for Spatiotemporal Deepfake Video Detection

Abstract

Now a days identification of convincing synthetic videos created with the help of deepfake is a big challenge. Unfortunately, deepfakes represent a serious threat to the integrity of media, as they can cause individuals to lose trust in the digital content they see. Among all types of deepfakes, face-swap videos are extremely difficult to detect. Deepfake videos usually have very high-quality and believable photos of facial features. The original article explains how researchers created a two-stage approach that detects deepfake videos by examining visual and motion-based information to be able to detect deepfake videos. The first stage uses a ResNet50 architecture for the creation of a convolutional neural network (CNN) with which to identify intrinsic visual artifacts caused by the manipulation of the videos through using the unique characteristics of the images and calculating how different types of video manipulations affect how people move through the frame of the video. The second phase utilizes the CNN-LSTM model to calculate temporal anomalies that occur between frames to determine if the movements of people as they appear in each frame were tampered with as well. In order to improve the ability of the above techniques to correctly identify deepfake videos, many additional changes were made, including background detection, filtering based on quality of the processed video, normalization of the processed video to be able to allow for any future changes, and extraction of necessary feature sets from the processed video. Experimental results demonstrate 98% accuracy and AUC 0.99 at video-level classifications, confirming that the integration of spatial and temporal information significantly outperforms approaches that rely on either aspect alone.