Machine Learning-Based Prediction And Long-Term Validation Of Colorectal Carcinoma Risk: A Generalizability Study Across Demographically Diverse Populations
Abstract
Colorectal carcinoma ranks third globally in cancer incidence and second in cancer-related mortality, with approximately 935,000 deaths recorded in 2020. Despite advancements in screening through colonoscopy and improved follow-up treatments, many patients continue to receive diagnoses at advanced stages, particularly across lower and middle-income regions. This study presents a machine learning system designed to estimate CRC likelihood, addressing two overlooked issues: absent extended time-frame testing and limited proof of broad applicability. A dataset of 5,430 patient records was assembled from 4 independent institutional cohorts and 6 demographic subgroups. Five classification algorithms were developed and systematically assessed: Logistic Regression, Naive Bayes, Support Vector Machine, Random Forest, and XGBoost. XGBoost achieved superior performance with an AUC-ROC of 0.961, accuracy of 94.3%, and F1-score of 0.939 on the primary test set. Longitudinal validation from 2018 to 2024 demonstrated that XGBoost maintained AUC-ROC values above 0.945 in each year, confirming sustained predictive durability. Validation across seven distinct populations — Caucasian, East Asian, South Asian, African, Hispanic, Middle Eastern, and others — yielded AUC-ROC scores ranging from 0.935 to 0.961, each exceeding the 0.90 minimal clinical threshold for model robustness. The framework provides both a high-performance predictive model and a repeatable methodological template for clinically validating ML models with demonstrated longevity.