Skip to content

The Value of System 1 Models in Medicine

Oct 2026 · 1 citation · 97 references
Computer Science

TL;DR

System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image, when it must be judged or read from an image.

Abstract

Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P<0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.

View source

Similar papers

Review Open access Oct 2026

Decision-only medical AI: early evidence and pairing with language models

Recently introduced decision-only AI returns probabilities over specified options rather than text. We tested the clinical consistency and potential bias susceptibility of the two available decision models, Jev and OpenAI Decisions, against a large language model (LLM), GPT-6 Luna, then paired each decision model with...

M. Omar, A. Ramaswamy, A. Sakhuja et al. · 0 citations
Open access Oct 2026

RadKev: An Open-Weight Decision Model for Radiology

Background: Many radiology decisions, such as whether a report describes a finding or how urgently a study should be read, are choices among a fixed set of answers. Decision models return a probability for each permitted answer in one step, without generating text, and an open-weight model can run within a hospital so...

U. Ram, R. Zhang · 0 citations
#artificial intelligence Review Oct 2026

A general-purpose System One decision model for clinical records

Clinical System One (CS1), a 4-billion-parameter decision model for fast, typed judgments from clinical records, is presented, along with two agentic harnesses in which CS1 works together with a generalist language model.

Andrew Zhang, Trinh Vuong, Sophia J. Wagner et al. · 1 citation
Preprint Oct 2026

From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

Large language models (LLMs) now answer medical exam questions at or above expert level, but it is unclear whether such scores reflect clinical competence, or are instead affected by the format of the questions being asked (e.g., multiple-choice versus open-ended), prior exposure to the benchmarks, and the difficulty o...

N. Nananukul, M. Kejriwal · 0 citations
#natural language process... Preprint Oct 2026

Detecting Spin in Clinical Trials with Large Language Models

Spin in clinical trials includes reporting practices that distort the presentation of results. This is particularly critical in medicine, where spin is present in more than 50% of randomized controlled trials that fail to reach statistical significance. The comparison of primary and reported outcomes is crucial for det...

Tjaša Ajdovec, Marko Robnik-Šikonja, Simon Suster · 0 citations
Review Open access Nov 2026

A decision-only model as a low-cost judge of medical AI answers

Background. Testing and deploying medical AI systems at scale already results in more outputs than clinicians can review, and the use of large language models (LLMs) as judges is growing rapidly. Decision-only models, cheap and fast models that return structured probabilities rather than text, offer a potentially scala...

M. Omar, Benjamin S. Glicksberg, A. Sakhuja et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.