A mechanistic intervention that reweighs attention away from the patch tokens in the identified lexical heads at test-time is performed and a test-time adaptive attention head routing is proposed to further reduce the impact on performance at matched robustness, pushing beyond the Pareto frontier of current robustness-perception tradeoff.
Abstract
CLIP-based Vision-Language Models have become cornerstones of modern multimodal systems. The cross-modal contrastive objective equips the CLIP vision encoder to both see objects and read text. These two abilities induce an entanglement between the encoding of lexical and semantic shapes, creating a distinctive vulnerability to typographic attacks, in which rendered text overrides an image's semantic content with its literal meaning. The black-box nature of these encoders has prevented a mechanistic interpretation of such entanglement. We introduce a head-decomposed Sparse Dictionary Learning (SDL) architecture that, by design, decomposes individual attention heads into interpretable, head-specific latents. Based on this inductive disentanglement, we re-purpose Clarity Score and propose normalized Text Attribution Score, which are both label-free, to identify attention heads that carry typographic information. As a causal validation of successful circuit separation, we perform a mechanistic intervention that reweighs attention away from the patch tokens in the identified lexical heads at test-time. Experiments show that the intervention effectively recovers object classification under typographic attack and, across five CLIP variants and four typographic benchmarks, achieves up to 46.9% improvement while incurring up to 1% accuracy loss in general image perception, outperforming prior supervised and training-free methods. Expanding on the mechanistic intervention, we propose a test-time adaptive attention head routing to further reduce the impact on performance at matched robustness, pushing beyond the Pareto frontier of current robustness-perception tradeoff.
Vision-language models such as CLIP enable zero-shot classification from arbitrary class names, but under test-time corruptions, their visual encoder projects distorted features onto incorrect textual anchors with high confidence: a phenomenon we term modality bias. Standard entropy minimization reinforces these overco...
Taehoon Kim, Wonjun Hwang, Hyunsouk Cho· 0 citations
EmbGA is presented, an embedding-space genetic algorithm that recovers concepts present in a set of images but absent from the prompt that generated them, and runs much faster than language-model-guided search.
While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment betw...
Sugam Karki, Raúl Vázquez, Yova Kementchedjhieva· Findings of the Association...· 0 citations
By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.
AI-generated images have reached unprecedented realism, while instruction-driven AI editors re-synthesize images for local changes, leaving no splice boundary to localize. Most detectors are black boxes with unverifiable verdicts and generalize poorly because adapting a backbone on the detection distribution memorizes...
Nikolaos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.