Linguistic Fidelity and Classification Performance of Large Language Models for Generating Synthetic Operative Notes: Evaluation Study

Abstract

Abstract Background Machine learning models for surgical applications require large, diverse datasets; however, data scarcity remains a critical limitation due to privacy regulations, institutional variability, and the rarity of many surgical procedures. Large language models (LLMs) offer a potential solution through synthetic data generation, but their performance and reliability in specialized surgical domains remain underexplored. Objective This study aimed to evaluate the linguistic fidelity of LLM-generated operative notes for cleft lip and palate procedures and to assess their impact on natural language processing classifier performance under varying data availability conditions. Methods A total of 630 authentic operative notes were obtained from cleft procedures (86 primary cleft lip repairs, 101 primary cleft palate repairs, and 62 primary alveolar bone grafting [ABG] procedures) performed between 2013 and 2024. GPT-4o generated matched synthetic notes using multishot prompting with anonymized examples. Linguistic fidelity was evaluated using BERTScore for semantic similarity, Jensen-Shannon divergence of part-of-speech trigrams for syntactic structure, and Bilingual Evaluation Understudy (BLEU) scores for lexical overlap. Binary classifiers using ClinicalBERT embeddings and logistic regression were trained under both full data and data-scarce (retaining 5% or 10% of positive training cases while retaining the full negative training set) conditions, with and without synthetic augmentation at approximate ratios of synthetic to real notes (1:1, 2:1, 5:1, and 10:1). Results Synthetic notes demonstrated high semantic fidelity across all procedures (BERTScore F1-score: 0.86‐0.88) and low syntactic divergence (Jensen-Shannon divergence: 0.06‐0.08). BLEU scores indicated moderate lexical variation (0.14‐0.19), reflecting distinct but contextually consistent phrasing. With full datasets, synthetic augmentation did not meaningfully affect classifier performance. Under data-scarce conditions retaining 5% of positive training cases while preserving the full negative training set, the area under the curve improved from 0.915 (SD 0.056) to 0.929 (SD 0.033) for cleft lip and from 0.935 (SD 0.036) to 0.949 (SD 0.033) for cleft palate, with smaller gains for ABG (mean 0.983, SD 0.014 to mean 0.987, SD 0.015). When retaining 10% of positive training cases, performance changes were minor across procedures (cleft lip: mean 0.952, SD 0.036 to mean 0.948, SD 0.028; cleft palate: mean 0.945, SD 0.044 to mean 0.956, SD 0.036; and ABG: mean 0.989, SD 0.008 to mean 0.985, SD 0.012). Conclusions LLM-generated operative notes exhibit strong semantic and syntactic fidelity to authentic documentation and can enhance model performance in a task-dependent manner when authentic data are limited. These findings suggest that synthetic data generation may address data scarcity challenges in specialized surgical domains, particularly for rare or underrepresented procedures, enabling robust machine learning model development.