Segment any tumour: an uncertainty-aware vision foundation model for whole-body analysis

Abstract

Prompt-driven vision foundation models, such as the Segment Anything Model, have shown adaptability in computer vision, but their use in medical imaging remains challenging due to heterogeneous anatomy, artefacts, and low-contrast tumour boundaries. This is particularly difficult in whole-body tumour analysis, where models must transfer across modalities, anatomies, and tumour appearances. Here, we present Segment Any Tumour 3D (SAT3D), a lightweight volumetric foundation model for generalisable tumour segmentation across diverse medical imaging modalities, organs, and cohorts. SAT3D integrates a shifted-window vision transformer with critic-guided uncertainty-aware training, using confidence maps as dense prompts to guide boundary prediction in ambiguous regions. We benchmark SAT3D against vision foundation models, prompt-driven and task-specific methods across 11 public datasets. Trained on 17,075 three-dimensional volume-mask pairs, SAT3D shows robust generalisation, including in out-of-distribution settings, and is supported by a 3D-Slicer plugin for interactive segmentation, underscoring SAT3D’s potential as a scalable foundation model for medical image analysis. The application of prompt-driven vision foundation models to medical imaging, particularly whole-body tumour analysis, remains challenging. Here, the authors develop Segment Any Tumour 3D (SAT3D), a lightweight volumetric foundation model for robust and generalisable tumour segmentation across diverse medical imaging modalities, organs, and cohorts.