Multimodal representation learning is critical for a wide range of applications, such as multimodal sentiment analysis. Current multimodal representation learning methods mainly focus on the multimodal alignment or fusion strategies, such that the complementary and consistent information among heterogeneous modalities can be fully explored. However, they mistakenly treat the uncertainty noise within each modality as the complementary information, failing to simultaneously leverage both consistent and complementary information while eliminating the aleatoric uncertainty within each modality. To address this issue, we propose a plug-and-play feature causality decomposition method for multimodal representation learning from causality perspective, which can be integrated into existing models with no affects on the original model structures. Specifically, to deal with the heterogeneity and consistency, according to whether it can be aligned with other modalities, the unimodal feature is first disentangled into two parts: modality-invariant (the synergistic information shared by all heterogeneous modalities) and modality-specific part. To deal with complementarity and uncertainty, the modality-specific part is further decomposed into unique and redundant features, where the redundant feature is removed and the unique feature is reserved based on the backdoor-adjustment. The effectiveness of noise removal is supported by causality theory. Finally, the task-related information, including both synergistic and unique components, is further fed to the original fusion module to obtain the final multimodal representations. Extensive experiments show the effectiveness of our proposed strategies.
Ye Liu, Zihan Ji, Hongmin Cai· Neural Information Processin...· 3 citations
Semi-supervised learning (SSL) aims to effectively utilize a small amount of labeled data together with a large volume of unlabeled data to improve learning performance. Among various SSL strategies, label propagation has been widely adopted due to its ability to diffuse label information across data points via graph structures. However, most existing label propagation-based SSL methods struggle with high-dimension-low-sample-size (HDLSS) data, as they rely on pairwise similarity measures that fail to capture the complex relationships among samples in such settings. To overcome this limitation, we propose a novel deep SSL framework that enhances label propagation using tensor-based similarity, enabling the modeling of high-order relationships among multiple samples. Specifically, we first pretrain a feature extraction network using the labeled data to obtain initial feature representations. Subsequently, tensor label propagation and fine-tuning of the feature extraction network are conducted iteratively. In the tensor label propagation module, pseudo-labels for the unlabeled samples are estimated more accurately by leveraging high-order similarity. These pseudo-labels, along with the original labeled data, are then used to fine-tune the pretrained network, resulting in more robust and discriminative feature representations across all samples. By embedding high-order structural information into the SSL pipeline, our method significantly enhances the prediction performance from limited labeled data in the HDLSS setting. Extensive experiments on multiple HDLSS datasets demonstrate the superiority of our approach compared to recent baselines.
Hongmin Cai, Jiali Sun, Fei Qi et al.· IEEE Transactions on Neural...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
