Shashwat Singh

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Vision-language (VL) models are typically built by fine-tuning a language model to attend to visual features in service of a Visual Question-Answering objective. We ask whether this alignment changes how language models represent text---especially on tasks where visualization would help, like geometry questions posed i...