A Multimodal Deep Learning Framework for Short-Video Content Classification and Automatic Tag Generation

Abstract

Short video clips from curated datasets combine rapidly changing visual, acoustic, and textual signals, yet existing methods treat classification and tag generation as separate tasks and degrade under missing modalities. This paper proposes MMTag-Net, a unified multimodal framework that jointly performs short-video classification and hierarchical tag generation. The framework integrates factorized video, audio-spectrogram, and BERT-based text encoders through bidirectional cross-modal attention, augmented by an entropy-based modality reliability gate, and employs a two-level class-conditioned decoder for coarse and fine-grained tags. Trained end-to-end with a multi-task loss combining classification, generation, and InfoNCE contrastive objectives, MMTag-Net achieves 85.2% top-1 accuracy on Kinetics-400 and 0.847 F1-score with 0.812 mAP on the curated SVMTD benchmark, outperforming X-CLIP and mPLUG-2 (p < 0.001) while processing 142 FPS with 118M parameters. The results confirm that reliability-gated multimodal fusion and joint task learning enable accurate, robust, and real-time semantic understanding of short video clips.