TL;DR
This work treats diffusion-based image generation as a sequential decision process and uses Diffusion Denoising Policy Optimization (DDPO) to learn a policy that steers generation away from a chosen target concept - in this case, "teddy bear."
Text-to-image diffusion models have achieved remarkable progress in image synthesis, but their potential misuse for generating unauthorized or harmful content has raised growing safety concerns. This has created an urgent need for safe diffusion-based image generation methods that can selectively suppress sensitive concepts while preserving the model’s general generative capability. Existing concept erasure approaches typically rely on either model fine-tuning or closed-form editing. However, they often suffer from two major limitations: (1 insufficient or excessive erasure, where the former fails to suppress target concepts and the latter disrupts benign semantics; and (2 degradation of non-target concepts, where removing target concepts undermines the generation of unrelated concepts, especially in multi-concept scenarios. To address these issues, we propose the Singular Value Eraser (SVEraser), a lightweight concept erasure module that removes specific concepts by optimizing singular-value offsets of weight matrices. Operating in a compact yet expressive singular-value space, SVEraser enables precise concept removal while reducing side effects on unrelated content. Moreover, once trained for different concepts, multiple SVErasers can be flexibly combined for multi-concept erasure. To further reduce interference, we introduce an eraser activation mechanism that adaptively selects the appropriate SVErasers during inference based on the input prompt. Extensive experiments on copyrighted objects, artistic styles, and explicit content demonstrate that our method achieves accurate target concept removal while preserving non-target semantics, providing a practical and reliable solution for safe diffusion-based image generation.
The results reveal that attention-space regulation offers a considerably more promising path to safer diffusion transformer based image generation than the existing concept erasing mechanism.
Modern text-to-image diffusion models rely on classifier-free guidance (CFG) to achieve high image fidelity and text alignment. However, CFG typically applies a static, global scale across all timesteps, samples, and conditions -- a choice that is generally suboptimal and can introduce artifacts, as different states may benefit from different levels of guidance. While time-varying schedules are known to improve quality, designing them by hand is non-trivial and application-dependent. In this paper, we learn the guidance schedule as a function of diffusion time, conditioning and the current noisy sample, in order to better align sampled images with the text prompt. We frame this as a density ratio estimation problem: a discriminator is trained to estimate the time-dependent log-density ratio between the true and guided marginal distributions, while a lightweight generator network predicts the optimal, state-dependent guidance scale. Empirically, our approach outperforms both heuristic CFG schedules and prior methods for learning dynamic guidance on text-to-image generation benchmarks.
Ensuring safety and policy compliance in text-to-image diffusion models remains a critical challenge, as benign or adversarial prompts can often elicit prohibited content, e.g. nudity and protected intellectual property. While training-based unlearning methods are effective, they are computationally expensive and prone to catastrophic interference with general capabilities. Conversely, existing test-time defenses are primarily prompt-centric, relying on modifying textual descriptions only, and overlook the visual signals for detection. In this paper, we propose to leverage the intermediate clean image estimated during the generation process and employ a sparse margin objective to detect prohibited concepts. When a violation is detected, we immediately intervene by optimizing a structured low-rank residual in the text-conditioning space via truncated backpropagation. This design allows weight-preserving detection, keeps non-violating inference latency nearly unchanged as the maximum budget increases, and offers flexibility in safety performance via test-time scaling. Extensive experiments on Stable Diffusion v1.4 and v3.5 across nudity removal, IP protection, and style erasure demonstrate superior performance across suppression, fidelity and preservation compared to prior weight-preserving baselines, providing a scalable and flexible solution for safe generative deployment.
This paper forms few-step generation as a controlled base generative process, and shows that self-consistency loss can be understood through the lens of optimal control, and draws a connection between this approach and reinforcement learning, potentially opening the door to a new set of approaches for few-step generation.
Uni-AdaVD is presented, a universal inference-time concept erasure framework for visual generation that treats the value space of multimodal attention as a unified intervention space and introduces encoder-aware target representation construction to localize target semantics across heterogeneous text encoders.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
