Why Only 3D for 3D? 2D-Guided Supervision for 3D Medical Vision–Language Models

Abstract

Training 3D medical vision--language models (VLMs) requires paired volumetric scans and expert annotations such as reports or segmentation masks, which are costly and limited. Meanwhile, large collections of unlabeled 3D scans and strong 2D medical VLMs are increasingly available. In this work, we study a practical paradigm: using pretrained 2D models as automatic annotators to supervise 3D VLMs. A 2D teacher model generates slice-level descriptions or masks that are aggregated into volume-level pseudo-labels to train a 3D student operating on full volumes at inference. We instantiate this 2D-to-3D supervision framework for report generation and segmentation. In label-scarce regimes, 2D-derived pseudo-labels substantially improve data efficiency. For report generation, pseudo-reports outperform or match ground truth when reports are limited and remain competitive at larger scales. For segmentation, pseudo-masks enable learning from unlabeled volumes and further improve performance when combined with limited expert masks. This data-centric perspective offers a practical path to scaling 3D medical VLMs by leveraging abundant 2D expertise as a scalable annotation source, without architectural changes. Because the teacher and student interact only through pseudo-labels, the framework also benefits automatically from stronger 2D models. Code: https://github.com/AlexQilong/2D-Guided-3D-Medical-VLM.