A CLIP-Based Multimodal Endorectal Ultrasound Model for Predicting Metachronous Liver Metastases in Rectal Cancer.

Objective

To develop and externally validate a contrastive language-image pretraining (CLIP)-based multimodal model integrating pretreatment multiplanar three-dimensional endorectal ultrasound (3D-ERUS) images and ERUS reports for predicting metachronous liver metastasis (MLM) after radical surgery in rectal cancer.

Methods

This two-center retrospective study included 394 patients with rectal cancer who underwent pretreatment 3D-ERUS within two weeks before treatment and subsequent radical surgery. Patients from center 1 were divided into training (n = 238) and internal validation (n = 102) cohorts, and center 2 served as an external validation cohort (n = 54). MLM occurred in 43, 19, and 5 patients, respectively. Sagittal, coronal, and transverse ERUS images and corresponding reports were encoded using Chinese-CLIP. After feature selection, balanced logistic regression models were developed, and a clinicopathological fusion model was constructed. Performance was assessed using AUC, Brier score, DeLong test, integrated discrimination improvement (IDI), and category-free continuous net reclassification improvement (NRI).

Results

The multimodal model achieved AUCs of 0.779, 0.824, and 0.743 in the three cohorts. The fusion model achieved higher AUCs of 0.812, 0.850, and 0.808. Compared to the clinical model, the fusion model significantly improved IDI and continuous NRI in the training cohort (IDI = 0.085; NRI = 0.647) and internal validation cohort (IDI = 0.114; NRI = 0.865). Compared to the multimodal model, the fusion model further improved IDI in both cohorts.

Conclusion

Chinese-CLIP-based integration of multiplanar 3D-ERUS images and reports showed potential for MLM prediction. Clinicopathological fusion may further improve risk stratification, warranting larger external validation.