Transferable Unrestricted Attacks via Holistic Inversion and Implicit Cross-Attention Regularization

Abstract

Pre-trained vision backbones, which are widely reused in automation and embodied perception systems, may be vulnerable to transfer-based adversarial attacks. A persistent challenge is to achieve strong transferability while preserving high perceptual fidelity of the adversarial examples. This work presents the Holistic and Detail-based Implicit Adversarial Attack (HDIAdv), an unrestricted attack framework built on a pre-trained text-to-image diffusion model to better balance these objectives. HDIAdv adopts a two-stage design. First, a holistic inversion module predicts an image-specific conditioning representation by estimating a placeholder-token embedding, which anchors diffusion sampling to the source content for high-fidelity generation. Second, an implicit cross-attention regularizer minimizes the spatial variance of U-Net cross-attention maps, discouraging excessive spatial concentration of placeholder-token attention during latent optimization. Experiments on an ImageNet-compatible benchmark evaluate both attack effectiveness and perceptual quality using Top-1 accuracy, FID, and LPIPS. Under the adopted evaluation protocol, HDIAdv achieves a favorable transferability-fidelity trade-off across diverse victim architectures and representative defense settings. Note to Practitioners—Perception pipelines in automation and embodied systems often reuse ImageNet-pretrained backbones, yet visually plausible adversarial inputs may transfer across architectures and remain effective after representative defenses. Our results indicate that robustness evaluations based only on small pixel-bounded perturbations may underestimate risks from unrestricted generative attacks. Practitioners should therefore include perceptually realistic, cross-model, and defense-aware stress tests when validating perception modules. The present evidence is limited to ImageNet-compatible classification and should not be interpreted as end-to-end validation of closed-loop robotic systems.