Heterogeneous Multimodal Fusion Using Deep Learning for Improved Feature Extraction and Benchmark Evaluation

Abstract

: Multimodal learning has emerged as a critical paradigm in artificial intelligence, enabling systems to integrate information from diverse data sources such as text, images, audio, and video. However, effectively fusing heterogeneous modalities remains challenging due to differences in feature dimensions, semantic representations, and temporal dynamics. This paper presents Hetero-Fusion-Net, a novel deep learning architecture that addresses these challenges through three key innovations: a Cross-Modal Attention Module for dynamic modality weighting, an Adaptive Fusion Layer for heterogeneous feature integration, and Joint Representation Learning for unified semantic space construction. We introduce the Multimodal Integration Score (MIS), a comprehensive evaluation metric that quantifies fusion quality through cross-modal consistency, prediction confidence, and classification accuracy. Extensive experiments on three benchmark datasets (CMU-MOSEI, IEMOCAP, and AV-MNIST) demonstrate that Hetero-Fusion-Net achieves 91.67% accuracy and an MIS of 0.8891, significantly outperforming early fusion (78.45% accuracy) and late fusion (81.23% accuracy, MIS: 0.7234) baselines. Our modality contribution analysis reveals optimal attention weights of 35% for text, 30% for images, 20% for audio, and 15% for video. The proposed architecture and evaluation framework provide a robust foundation for advancing multimodal fusion research across diverse application domains.