Skip to content
Preprint

Label-Free Localization and Test-Time Suppression of CLIP's Typographic Circuits

Nov 2026 · 0 citations · 29 references
Computer Science

TL;DR

A mechanistic intervention that reweighs attention away from the patch tokens in the identified lexical heads at test-time is performed and a test-time adaptive attention head routing is proposed to further reduce the impact on performance at matched robustness, pushing beyond the Pareto frontier of current robustness-perception tradeoff.

Abstract

CLIP-based Vision-Language Models have become cornerstones of modern multimodal systems. The cross-modal contrastive objective equips the CLIP vision encoder to both see objects and read text. These two abilities induce an entanglement between the encoding of lexical and semantic shapes, creating a distinctive vulnerability to typographic attacks, in which rendered text overrides an image's semantic content with its literal meaning. The black-box nature of these encoders has prevented a mechanistic interpretation of such entanglement. We introduce a head-decomposed Sparse Dictionary Learning (SDL) architecture that, by design, decomposes individual attention heads into interpretable, head-specific latents. Based on this inductive disentanglement, we re-purpose Clarity Score and propose normalized Text Attribution Score, which are both label-free, to identify attention heads that carry typographic information. As a causal validation of successful circuit separation, we perform a mechanistic intervention that reweighs attention away from the patch tokens in the identified lexical heads at test-time. Experiments show that the intervention effectively recovers object classification under typographic attack and, across five CLIP variants and four typographic benchmarks, achieves up to 46.9% improvement while incurring up to 1% accuracy loss in general image perception, outperforming prior supervised and training-free methods. Expanding on the mechanistic intervention, we propose a test-time adaptive attention head routing to further reduce the impact on performance at matched robustness, pushing beyond the Pareto frontier of current robustness-perception tradeoff.

View source

Similar papers

#machine learning Preprint Nov 2026

Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation

Vision-language models such as CLIP enable zero-shot classification from arbitrary class names, but under test-time corruptions, their visual encoder projects distorted features onto incorrect textual anchors with high confidence: a phenomenon we term modality bias. Standard entropy minimization reinforces these overco...

Taehoon Kim, Wonjun Hwang, Hyunsouk Cho · 0 citations
Open access Oct 2026

Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers

While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment betw...

Sugam Karki, Raúl Vázquez, Yova Kementchedjhieva · 0 citations
#artificial intelligence Preprint Nov 2026

Generalizable AI-Generated Image Detection with Embedding-Space Explanations

AI-generated images have reached unprecedented realism, while instruction-driven AI editors re-synthesize images for local changes, leaving no splice boundary to localize. Most detectors are black boxes with unverifiable verdicts and generalize poorly because adapting a backbone on the detection distribution memorizes...

Nikolaos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.