Although Large Language Models (LLMs) have demonstrated promising safety performance, extending them to Multimodal Large Language Models (MLLMs) exposes a significant gap between expanded multimodal capabilities and existing safety mechanisms. Current defenses remain predominantly confined to specific modal settings, thereby limiting their robustness against broader cross-modal threats. To bridge this gap, we introduce SafeNexus, a cross-modal safety alignment framework that adopts a dedicated neuron-level intervention strategy. First, we formulate a neuron localization paradigm that identifies functionally specialized neurons by characterizing intermediate-layer activation patterns and quantifying their functional salience through importance scoring. Building upon this paradigm, we exploit contrastive data to identify modality-bound safety neurons (BS-Neurons), and validate their role in regulating safety behavior within each modality via targeted suppression. Further cross-modal analysis defines modality-universal safety neurons (US-Neurons) as the shared subset of BS-Neurons identified across individual modalities, serving as the core for defending against harmful cross-modal attacks. We observe that suppressing these neurons substantially degrades safety performance across modalities, while leaving overall utility largely unaffected. Building on these insights, we propose two safety alignment strategies: activation-level safety amplifier and safety neuron calibrator. The proposed strategies enhance model safety through two distinct routes: the former amplifies the activation magnitudes of US-Neurons, while the latter selectively calibrates them via targeted fine-tuning. Extensive experiments demonstrate that our method outperforms prevailing state-of-the-art approaches on safety benchmarks spanning diverse modality combinations, while effectively preserving utility.
As a paradigm in continual learning, class incremental learning (CIL) aims to assimilate tasks with mutually exclusive label spaces in sequence while preserving previously established knowledge. Mitigating forgetting in CIL fundamentally relies on transferring knowledge across tasks. A straightforward exemplar-based approach promotes balanced knowledge transfer by replaying an equal number of samples from each old class. However, in the more challenging exemplar-free setting, this balance cannot be ensured because distillation-based cross-task knowledge transfer tends to focus more heavily on the knowledge acquired from the most recent tasks. To address the unfairness in knowledge transfer, we analyze the mechanisms underlying dark knowledge and introduce a Semantic Enhanced Knowledge Transfer (SEKT) method for exemplar-free CIL. Specifically, SEKT adopts a bi-flow framework. The first flow is the Semantic Guidance Flow (SGF), which is inspired by knowledge distillation and produces latent semantic distributions from the outputs of the old model to guide the new model toward generating similar distributions. The second flow is the Semantic Propagation Flow (SPF), which propagates latent early knowledge to the current task in order to mitigate the unfairness in knowledge transfer. SPF constructs a cross-task semantic similarity graph using aligned intermediate representations to enable semantic propagation. It employs an expert network to learn the pattern of semantic propagation, enabling real-time and stable semantic recovery during training. In contrast to the SGF that is more effective for transferring recent knowledge, the SPF learns complementary early knowledge through a semantic complementarity constraint. Moreover, the SPF is robust to noisy semantics, as the learned semantic distribution is regularized with an $\ell _{2,1}$ norm. Extensive experiments conducted on six datasets demonstrate the superiority of the proposed SEKT over existing exemplar-free CIL approaches.
Fan-Kang Xu, Lu Jin, Yanpeng Sun et al.· IEEE Transactions on Image P...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
