Brian Diep

Interpreting multimodal speech-vision models with sparse autoencoders

Vision-language models are a form of multimodal AI that encode language and visual information jointly for a range of downstream tasks, however interpretation of how they understand the input across modalities remains difficult. This work uses Sparse Autoencoders (SAEs) to interpret a self-supervised audio-visual model...