Real‐world clinical impact of implementing and updating a deep learning‐based automatic contouring system in rectal cancer radiotherapy

Abstract

Abstract Background Accurate delineation of target volumes and organs‐at‐risk (OARs) is a critical yet labor‐intensive component of rectal cancer radiotherapy. While deep learning (DL)‐based automatic contouring systems are increasingly used to address inter‐observer variability and improve efficiency, artificial intelligence models require rigorous quality assurance and updates to reflect current technology. However, high‐level evidence regarding the longitudinal real‐world impact of implementing and iteratively updating these systems in clinical workflows is currently lacking. Purpose This study aimed to evaluate the real‐world clinical impact of implementing and updating a DL‐based automatic contouring system in rectal cancer radiotherapy to generate high‐quality evidence of iterative updates. Methods This longitudinal retrospective analysis included 150 patients divided into three cohorts: pre‐implementation (n1 = 50), post‐implementation (n2 = 50), and post‐update (n3 = 50). Geometric similarities between unedited‐automatic and final treatment contours were compared across cohorts. Failure rates were systematically analyzed. Six oncologists contoured 21 additional cases through manual, first‐generation (Auto1), and second‐generation (Auto2) system‐assisted methods to evaluate contouring time, inter‐observer consistency, and accuracy. Additionally, a 5‐point Likert scale was used by two blinded senior oncologists to assess the clinical acceptability of the generated contours. Results The mean Dice similarity coefficient (DSC) values of clinical target volume (CTV) before and after implementing the automatic contouring system were 0.87 ± 0.04 and 0.88 ± 0.04 (P = 0.067), while those of OARs were 0.80 ± 0.06 and 0.88 ± 0.05 (P < 0.001), respectively. Following the system update, they improved from 0.88 ± 0.04 to 0.93 ± 0.04 for CTV (P < 0.001) and from 0.88 ± 0.05 to 0.95 ± 0.02 for OARs (P < 0.001). The system update achieved an approximately 80.6% reduction in the mean failure rate. Auto2‐assisted method decreased the total time by approximately 58.8% compared with the manual method, and 21.9% compared with the Auto1‐assisted method. This method also demonstrated optimal inter‐observer consistency (0.95 ± 0.03) and accuracy (0.94 ± 0.03) for CTV. In the blinded clinical evaluation, 99.2% (125/126) of the oncologist‐revised final contours received a Likert score of ≥ 4, and Auto2‐generated unedited contours showed significantly higher clinical acceptability than Auto1 (4.02 ± 0.25 vs. 3.26 ± 0.49, P < 0.001) Conclusions Implementing an automatic contouring system provided crucial guidance for clinical practice. Its iterative update significantly reduced workload and inter‐observer variation while enhancing contouring efficiency and quality.