Structure-Level Concept Erasure via Self-Attention Disruption in Diffusion Models

Abstract

Text-to-image diffusion models can synthesize unsafe content on demand, which makes concept erasure an active safety problem. Existing training-free methods all intervene on the model's content pathway: they steer the predicted noise, project the text embedding, or rewrite cross-attention weights, controlling what the model paints. Yet a visual concept is carried not only by content but by structure, the way spatial parts bind into a coherent whole, which in diffusion U-Nets is governed by self-attention. We introduce SADE (Self-Attention Disruption Erasure), a training-free method that erases a concept by disrupting its structural assembly rather than its content: it reads a spatial saliency for the concept from the self-attention features, then down-weights the mutual attention among salient positions so the concept cannot form. The operator changes no weights, adds one strength parameter, and provably reduces to ordinary generation when the target concept is absent. On I2P, SADE matches state-of-the-art training-free erasure while preserving COCO image quality best among the compared methods, and its prompt-independent saliency resists text-based attacks. Existing methods stop the model from painting the concept; SADE lets it paint but keeps the pieces from coming together.