Large language models to support oncology symptom telephone triage: A retrospective pre-implementation evaluation.

Abstract

672 Background: Oncology symptom telephone triage is an essential supportive care service for patients with cancer, providing assessment and guidance for symptoms outside scheduled visits. Rising call volumes and clinical complexity create a need for tools that improve timeliness, consistency, and documentation quality without compromising safety. We evaluated whether large language models (LLMs) can support these workflows by generating acuity classifications concordant with expert assessment and high-quality patient- and physician-facing response drafts. Methods: We retrospectively evaluated 153 real-world oncology symptom telephone triage cases from the Cancer Centre of Southeastern Ontario, a Canadian academic cancer center. Prompts incorporated patient and cancer characteristics, treatment details, and symptom information available to triage staff in routine practice. All cases were de-identified before model input. ChatGPT 5.5, Gemini 3.5, Copilot, and three clinical trainees (one PGY4 medical oncology fellow and two senior medical students) independently generated acuity classifications, patient-facing counselling, and physician-facing documentation drafts. Acuity concordance was assessed against an expert-adjudicated reference standard set by two oncologists, with discordant ratings adjudicated by a third senior oncologist. Oncologists rated response quality for empathy, clinical accuracy, usefulness, and safety on 5-point scales (maximum composite 20). Response generation time was recorded. Results: Acuity concordance was comparable across LLMs, 71.4%-73.4%, and trainees, 61.0%-69.5%. Among discordant cases, LLMs more often over-triaged, including ChatGPT in 97.6%, Copilot in 95.5%, and Gemini in 83.0%, reflecting a safety-favoring pattern. Trainees more often under-triaged, with under-calls in 55%-60% of discordant cases. No trainee outperformed LLMs on composite quality. The oncology fellow performed closest to LLMs with concordance of 69.5% and composite scores of 16.84/20 patient-facing, and 16.64/20 for physician-facing drafts, yet every LLM matched or exceeded all trainees while generating higher-rated drafts substantially faster. Copilot had the highest composite scores for patient-facing responses, 18.03/20, and physician-facing drafts, 18.40/20. LLM response times were 126-130 seconds compared with 209-454 seconds for trainees. Conclusions: In this real-world oncology symptom triage dataset, LLMs performed at or above the oncology fellow and senior medical students on acuity concordance, showed a safety-favoring over-triage pattern, and produced higher-rated response drafts faster than trainees. These findings support prospective evaluation of LLMs as staff-supervised flagging and drafting tools to improve efficiency, consistency, and documentation quality within oncology symptom triage workflows.