CMJU: cross-modal joint unlearning for balanced forgetting in multimodal large language models

Abstract

Multimodal large language models (MLLMs) may memorize private or sensitive knowledge from training data, making unlearning important for safe deployment. Existing studies have shown that unlearning in MLLMs can remain inconsistent across multimodal and text-only inputs: target knowledge that has been forgotten through one input modality may still be retrieved through the other. Recent benchmarks provide both multimodal and text-only forgetting samples, but using them together does not guarantee balanced unlearning across the two input forms. We study this problem under the modality-complete supervision setting and propose cross-modal joint unlearning (CMJU), a unified framework for balanced cross-modal forgetting. CMJU first estimates parameter saliency along the text-only and multimodal paths, and then selects active parameters and groups them by modality preference. Next, it performs joint unlearning with coordinated update mechanisms, including balanced gradient updates, shared group scaling, and a symmetry constraint, which allow forgetting to proceed more coordinately across modalities while preserving non-target knowledge. Experiments on benchmarks including UMU-Bench and CLEAR show that CMJU achieves better cross-modal forgetting balance than strong baselines, while maintaining competitive retain-set performance. These results suggest that balanced unlearning in MLLMs does not automatically occur with modality-complete supervision. Instead, it requires explicit coordination between multimodal and text-only unlearning signals.