From pixels to actions: grounding in vision-language models

Abstract

Vision-Language Models (VLMs) have demonstrated impressive capabilities across a broad range of perception tasks, yet systematic weaknesses remain concerning fine-grained spatial reasoning, precise visual localisation, or grounded action in interactive environments. This thesis investigates the architectural and training decisions that govern these capabilities through controlled experiments, systematic architecture comparisons, and model development. We first examine whether feature-compressing connectors, i.e., subnetworks that downsample visual inputs for the language decoder, preserve the spatial information and show that these networks do not retain spatial precision, but this behaviour is sensitive to the training objectives rather than an architecture bottleneck. Second, we explore language model alternatives for efficiently modelling long multimodal sequences by comparing capacity and data-matched Transformer and State Space Model (SSM) based decoders across coarse and fine-grained multimodal tasks, showing that SSMs are competitive when outputs rely on image summarisation but struggle when explicit retrieval from context is required. The third contribution addresses this retrieval deficit through a controlled study of hybrid Transformer–SSM architectures, demonstrating that hybrid models match or exceed Transformers in data efficiency and length extrapolation on retrieval-demanding tasks. The remaining contributions of the thesis focus on applications to interactive tasks that require grounded action for sequential decision making. The first setting involves the development of a VLM pretrained on static image-text pairs and adapted for embodied task completion within a simulated 3D environment. Finally, we focus on improving the action execution component of a planner-enabled Graphical User Interface (GUI) agent. We show that a carefully curated dataset of demonstrations augmented with chain-of-thought traces and reinforcement learning with sparse verifiable rewards yields a compact model that matches or exceeds agents trained on substantially larger corpora. Together, these contributions provide empirical evidence of where and why current VLMs fall short of fine-grained and interactive understanding, and identify concrete strategies for addressing these limitations.