Interpreting multimodal speech-vision models with sparse autoencoders

Abstract

Vision-language models are a form of multimodal AI that encode language and visual information jointly for a range of downstream tasks, however interpretation of how they understand the input across modalities remains difficult. This work uses Sparse Autoencoders (SAEs) to interpret a self-supervised audio-visual model, focusing on spoken language rather than text as the linguistic modality. We train and compare four SAE architectures (Standard, JumpReLU, BatchTopK, and Matryoshka) on audio-only, image-only, and joint embeddings, and evaluate them along a sparsity-fidelity tradeoff. Qualitative analysis shows that individual SAE dimensions recover concepts at multiple levels of granularity, from single words and semantic categories to phrase-level and position-sensitive structure in the audio stream. Some dimensions capture near-singular lexical items, while others generalize across semantically or phonetically related words. Audio-focused SAEs are consistently less monosemantic than image-focused ones, and Matryoshka models show lower average monosemanticity than JumpReLU models, indicating greater feature entanglement. To test whether concepts are causally shared across modalities, we introduce an image-attenuated word segmentation task: suppressing an SAE dimension in the image branch and propagating this intervention to the audio embedding. Ablating a dimension in the image modality degrades the localization of the corresponding spoken word relative to controls, but the effect is small unless the analysis is restricted to the subset of dimensions with genuinely multimodal activation, at which point the effect becomes substantially clearer and more consistent, particularly for JumpReLU models. Audio ablation paradoxically produces little change in Matryoshka models, which we attribute to hierarchical entanglement across dimensions. These results indicate that our studied VLMs develop some modality-agnostic conceptual representations through unsupervised statistical learning, but that a persistent modality gap separates audio and image encodings of the same concept.