Skip to content

Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs

Aug 2026 · arXiv.org · Vol abs/2608.24354 · 0 citations · 60 references
Computer Science

TL;DR

RACER is presented, a model-level repair framework motivated by a key observation: backdoors induce abnormal layer-to-layer evolution in internal representations, which is modality-dependent, concentrating primarily in the token region encoding the trigger features that the backdoor model actually relies on.

Abstract

MLLMs are increasingly deployed in user-facing applications, yet they inherit backdoor risks from the pipelines used to construct them: triggers may reside in images, texts, or both. Existing model-level backdoor removal methods, largely designed for conventional classifiers, show limited effectiveness on MLLMs, while MLLM-specific defenses mainly operate at inference time, filtering suspicious inputs without removing the backdoor embedded in the model. To address this gap and eliminate latent backdoors from MLLMs at their source, we present RACER, a model-level repair framework motivated by a key observation: backdoors induce abnormal layer-to-layer evolution in internal representations, which we term the layer-wise inconsistency anomaly. Importantly, this anomaly is modality-dependent, concentrating primarily in the token region encoding the trigger features that the backdoor model actually relies on. RACER therefore decomposes the fused representation into visual and textual token regions, normalizes their layer-wise inconsistency separately, and recomposes them using modality-aware weights over a deep-layer window, yielding a region-aware inconsistency objective that better captures localized backdoor-induced anomalies. Through a min-max optimization, this objective drives worst-case perturbation synthesis and adversarial fine-tuning against the resulting perturbation to repair the model, suppressing the deep representational directional shifts on which backdoor behaviors rely. RACER requires only 100 clean samples and no knowledge of the trigger, attack objective, or even whether the input model contains a backdoor. Evaluations on three open-source MLLMs across 36 backdoor settings spanning image, text, and multimodal triggers show that RACER reduces the average ASR to 1.1%, reaching 0% in 32 settings, while preserving clean-task utility on both backdoor and clean models.

View source

Similar papers

#machine learning Preprint Oct 2026

Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse

Traditional backdoor attacks inject artificial trigger patterns into training data, causing models to misclassify triggered inputs while maintaining normal behavior on clean samples. Since these artificial triggers are absent in clean datasets, standard knowledge distillation (KD) typically eliminates them, leading to...

Junhoo Lee, Baekseung Kim, Seungyeon Kim et al. · 0 citations
Preprint Oct 2026

SemPurify: Semantics-Aware Data Purification for Black-box Backdoor Defense

This work proposes the first semantics-aware data purification framework for black-box backdoor defense, SemPurify, which explicitly incorporates multi-level semantics within a text-to-image (T2I) framework to guide image restoration during purification.

Jia-Wei Yu, Syed Irfan Ali Meerza, Yi Wu et al. · 0 citations
#small language model Preprint Oct 2026

Purifying Backdoored Large Vision-Language Models by Removing Hijacked Directions

OrthoPurify is proposed, a more efficient method to purify backdoored model weights via one-step orthogonal projection, which reduces the attack success rate to near zero while preserving the original performance across diverse benchmarks, without retraining the backdoored model or introducing inference-time overhead.

Bo Yang, Hao Zhou, Zhi-Fang Zhang et al. · 0 citations
Preprint Oct 2026

TMT: Runtime Backdoor Detection for Vision-Language-Action Policies on Unseen Tasks

Backdoored vision-language-action (VLA) policies can preserve benign task performance while producing malicious actions when a trigger appears. Detecting such activation is difficult because malicious behavior can comprise individually plausible actions, while unfamiliar tasks introduce legitimate changes in observatio...

Zi-Run Zhou, Jing-Feng Zhang, Hao-Chuan Xu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Towards Merging-conditional Backdoor Unalignment against Large Language Models

Model merging has become widely adopted for enhancing the task-specific performance of Large Language Models (LLMs) without additional training. While the specialized model from third-party platforms may undergo safety audits (e.g., backdoor detection), we reveal a novel and critical threat: One-sided Merging-Condition...

Yu-Kun Chen, Yi-Dou Chen, Yi-Ming Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Backdoor Containment via Expert Quarantine and Shutdown in LLMs

Backdoored large language models (LLMs) can behave normally on benign inputs while producing attacker-specified outputs under hidden triggers. Existing defenses span four stages---prior-training, in-training, post-training, and inference-time---and share one of two underlying strategies: either suppress backdoor learni...

Jian-Wei Li, Min-Seon Kim, Jung-Eun Kim · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.