Background Generative Artificial Intelligence (GAI) models, such as GPT-4, have been extensively studied for their integration into medical practice and education. GPT-4 has demonstrated excellent performance on medical licensing examinations, including the United Kingdom Medical Licensing Assessment (UKMLA). However, the field lacks longitudinal data on whether such performance is stable or varies over time. Theoretically, iterative improvements updated by the vendor should enhance performance, but empirical evidence of such longitudinal trends remains limited. Given that GPT-4 undergoes periodic vendor-side updates, we aimed to analyse the categorical, time-spaced performance of GPT-4 on the UKMLA to characterise how its performance changes over time in a medical context. Methods Two publicly available UKMLA papers were fed into GPT-4 at two different time points, June 2023 and December 2024. 191 questions were provided with and without multiple-choice options to assess GPT-4’s clinical competence. McNemar’s test was performed to evaluate changes in GPT-4’s performance over time, comparing domain-specific questions. Results GPT-4’s accuracy improved noticeably between the two rounds (MCQ: 88.0 to 93.7%, p = 0.027; non-MCQ: 68.1 to 81.7%, p = 1.00). Single-step accuracy rose from 73.1 to 82.3%, and multi-step from 57.4 to 80.3% without MCQ. GPT-4 showed improved accuracy from Round 1 to Round 2 for both single-step and multi-step questions, with MCQ-prompted responses consistently outperforming non-MCQ responses (up to 95.1% accuracy for multi-step MCQ questions in Round 2). GPT-4’s performance improved across all question categories from round one to round two, most notably in management questions without MCQ options (+23.30%), though these differences were not statistically significant. Discussion and conclusion GPT-4’s performance on the UKMLA improved significantly over 18 months, suggesting that iterative vendor-side model updates enhance clinical reasoning capabilities. These findings indicate that GPT-4 may serve as a supplementary educational tool for medical students and clinicians; however, the underlying drivers of performance changes remain opaque, and such tools should be deployed with structured oversight to prevent overreliance.
Ravanth Baskaran, Sai Sirikonda, Aditya Singh et al.· Frontiers in Medicine· 0 citations
Background Mutation-derived neoantigens, typically identified in primary tumors, are emerging therapeutic targets for personalized cancer vaccines and adoptive T-cell therapies. However, clinical efficacy of neoantigen-directed therapies in patients with metastatic disease remains limited, partly due to inter-site genetic heterogeneity. We investigated whether ubiquitous neoantigens–derived from mutations shared across all tumor sites–could provide more effective, durable targets, particularly in patients undergoing resection of metastatic lesions. Methods Whole-exome and RNA sequencing were performed on 14 tumor samples (primary and 13 synchronous nodal metastases) from a treatment-naïve patient with pancreatic neuroendocrine tumor (PNET). Ubiquitous mutations were identified bioinformatically, and their immunogenicity assessed using in-vitro stimulation of autologous peripheral blood mononuclear cells followed by IFN-γ ELISpot assay. Neoantigen-specific T-cell clonotypes were further identified by HLA-tetramer staining and single-cell RNA/TCR sequencing. Neoantigen-reactive clonotypes identified in peripheral blood were tracked across multiple metastatic sites using bulk TCRβ repertoire sequencing. Results Among 1,195 non-synonymous mutations detected, eight were shared across all 14 tumor sites. Of these, one encoded a neoantigen that elicited a reproducible IFN-γ ELISpot response in peripheral blood, confirming its immunogenicity. Further, we identified the corresponding neoantigen-reactive TCR clonotypes in blood. Comparison with bulk TCRβ repertoires from eight metastatic sites showed that these clonotypes were present in every site analyzed, with evidence of local clonal expansion. Conclusion This study provides direct evidence that a single ubiquitous mutation-derived neoantigen can generate systemic T-cell responses and clonotype expansion across multiple metastatic sites in a TMB-low, TIL-low tumor. Our findings support incorporating mutation-sharing status across metastases as a key criterion for neoantigen selection in cancer vaccines and adoptive T-cell therapies. This approach could inform the design of neoantigen-directed immunotherapies in metastatic PNET and potentially other metastatic solid tumors. What is already known on this topic Neoantigen-directed therapies, such as personalized cancer vaccines or adoptive T-cell transfer, can induce anti-tumor responses but have shown limited success in metastatic disease. One major barrier is genetic heterogeneity between tumor sites, suggesting that targeting ubiquitous mutations–those shared across all tumor sites–may improve the efficacy of such therapies. What this study adds In one patient with metastatic pancreatic neuroendocrine tumor involving 13 lymph nodes, we identified eight ubiquitous mutations, one of which generated a detectable neoantigen-specific T-cell response in blood. The corresponding T-cell clonotypes were found across all metastatic sites analyzed and showed evidence of clonal expansion, providing direct evidence of systemic and local recognition of a shared neoantigen in a TMB-low/TIL-low cancer. How this study might affect research, practice or policy These findings support incorporating mutation sharing across metastases as a key criterion in neoantigen selection for cancer vaccines and adoptive T-cell therapies. This strategy could enhance the relevance and durability of neoantigen- directed approaches in patients with metastatic disease.
J. Tanis, Katy J. McCann, F. E. Castañeda-Castro et al.· bioRxiv· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
