Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers
While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment betw...