Skip to content

Large Language Models vs. NASS Common Coding Scenarios: An Analysis of Accuracy and Financial Implications using 2026 CPT AMA Codebook Guidelines.

Aug 2026 · The spine journal · 0 citations · 13 references
Medicine

TL;DR

This study evaluates baseline performance under isolated zero-shot conditions; the tendency of Claude and ChatGPT to create unbundled billable codes presents a severe audit risk, while Gemini demonstrated a lower mean financial variance.

Abstract

Background

Context

Accurate Current Procedural Terminology (CPT) coding is essential for compliant revenue cycle management in spine surgery. However, increasing documentation burdens and the complexity of bundling rules often lead to significant revenue leakage or audit risk.

Purpose

To evaluate the baseline efficacy of three state-of-the-art Large Language Models (LLMs) in generating accurate CPT codes compared to the "Gold Standard" North American Spine Society (NASS) Common Coding Scenarios, and to quantify the downstream financial impact. STUDY DESIGN/

Setting

Comparative artificial intelligence performance analysis. PATIENT SAMPLE Twenty standardized clinical vignettes selected from the NASS Common Coding Scenarios guide, representing a broad spectrum of spine pathology including cervical decompression/fusion, lumbar decompression, lumbar fusion, and complex deformity. OUTCOME MEASURES Exact match accuracy, error of omission (under-coding), error of commission (over-coding/unbundling), and net financial variance utilizing 2026 Medicare Physician Fee Schedule (MPFS) Work RVU (wRVU) values mapped to a standardized commercial conversion factor of $60/wRVU.

Methods

Each vignette was input into three LLMs: GPT-4o (OpenAI), Gemini 1.5 Pro (Google), and Claude 4.6 Sonnet (Anthropic). Under isolated zero-shot conditions, the models were prompted to function as certified professional coders and generate the appropriate CPT codes and modifiers. Outputs were scored against the official NASS answer key, and subsequent financial variances were calculated.

Results

Gemini 1.5 Pro achieved an exact match rate of 65%, followed by ChatGPT-4o (55%) and Claude 4.6 Sonnet (40%), though these differences in overall performance were not statistically significant (p = 0.28). However, performance varied substantially by procedure type; notably, both ChatGPT and Claude achieved a 0% exact match rate for Lumbar Fusion constructs, frequently failing to correctly bundle interbody and posterior instrumentation codes. Financially, Gemini 1.5 Pro was the most stable, resulting in a negligible mean net revenue variance of -$2.16 per case. In contrast, ChatGPT and Claude demonstrated a strong tendency toward over-coding and high financial volatility, resulting in mean net revenue overcharges of +$75.60 and +$104.07 per case, respectively.

Conclusions

While no statistically significant difference in overall accuracy was detected between models, qualitative deficits remain severe in handling complex fusion hierarchies. This study evaluates baseline performance under isolated zero-shot conditions; the tendency of Claude and ChatGPT to create unbundled billable codes presents a severe audit risk, while Gemini demonstrated a lower mean financial variance. Currently, LLMs should function only as adjunctive tools requiring strict human supervision.

View source

Similar papers

Oct 2026

Readability, CPT Billing Accuracy and Diagnostic Coding Performance of ChatGPT Models in Hand Surgery.

Background: Large language models (LLMs) like ChatGPT show their potential use in healthcare. However, limited research has been conducted on how effective these models are in hand surgery. Therefore, the purpose of this study is to compare ChatGPT-4, ChatGPT-4o and ChatGPT-o1 across patient education, billing and diag...

S. Daulat, Romir P. Parmar, Tyler T. Brady et al. · 0 citations
Review Open access Sep 2026

Evaluating reasoning-tuned large language models for clinical decision-making in spine surgery.

PURPOSE Most clinical evaluations of large language models assess factual recall rather than the multi-step reasoning behind operative plans. Reasoning-tuned models, post-trained to generate explicit intermediate reasoning before answering, may better approximate surgical decision-making. We compared two such models fr...

C. Lam, Conor T. Boylan, Adit Ravishankar et al. · 0 citations
Review Sep 2026

Large Language Model Data Abstraction Demonstrates Accuracy and Reliability for NSQIP.

BACKGROUND National Surgical Quality Improvement Program (NSQIP) data collection depends on labor-intensive manual chart abstraction, limiting efficiency, increasing cost, and necessitating patient sampling. This study evaluated whether a large language model (LLM) could accurately abstract unstructured NSQIP breast re...

Tahera Alnaseri, Yasmine Ibrahim, Ansgar Grunseid et al. · 0 citations
#natural language process... Preprint Sep 2026

Decomposing Error and Style in Automated Clinical Coding

This work model the systematic component as coding style $\psi$, a coder- or site-specific policy over what to code and how much to document, and recast coding as $p(\mathrm{code}\mid\mathrm{note},\psi)$, estimating $\psi$ with a 10-dimension rubric.

Han-Chin Shing, Jack Moriarty, Ryan Ware et al. · 0 citations
Oct 2026

Assessment of accuracy, quality, and readability of large language models in kidney transplantation practice

Current-generation LLMs show high and comparable performance in addressing complex KT-related clinical questions and their consistent accuracy, quality, and readability support their potential role as adjunctive clinical decision-support tools, provided appropriate physician oversight.

M. Polat, Mecit Çelik, Kazım Ceviz et al. · 0 citations
Aug 2026

Evaluation of Large Language Model-Generated Recommendations in Glaucoma Surgical Decision-Making.

LLM-based chatbots can provide acceptable surgical guidance for straightforward primary surgical cases, but their utility is limited in high-risk or complex clinical settings, and their rationale quality was significantly lower in complex scenarios than in primary ones.

Müge Toprak, Büşra Yılmaz Tuğan, N. Yüksel · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.