It is suggested that language models for high-stakes applications should be understood and evaluated as probabilistic decision systems whose recommendations depend jointly on predictive performance and explicit utilities.
Abstract
High-stakes decisions under uncertainty, such as medical emergency triage, require more than accurate predictions. They depend on estimating the likelihood of alternative outcomes while explicitly weighing the consequences of different actions, principles that have long formed the foundation of medical diagnosis and decision making. Yet language models are increasingly used for high-stakes clinical recommendations without explicit specification of the utilities governing these decisions. Here we show that emergency triage with language models can be understood within a probabilistic decision framework, providing a case study of a broader decision-analytic paradigm for steering, evaluating, and deploying language models in high-stakes settings. Using clinical vignettes from a structured evaluation of a consumer triage system, we analyze recommendations for treatment under alternative utility functions that specify the relative costs of missed emergencies and unnecessary escalation. We find that capable language models adjust recommendations in response to stated utilities, revealing that the same underlying predictions can support markedly different decision policies. These findings show that effective deployment depends not only on improving predictions but also on making decision objectives explicit. More broadly, they suggest that language models for high-stakes applications should be understood and evaluated as probabilistic decision systems whose recommendations depend jointly on predictive performance and explicit utilities.
This work presents a benchmark of 208 clinically grounded rare disease vignettes and identifies a strong authority-framing effect: models favor justice in committee-based contexts and shift toward beneficence and autonomy only when final decisions are framed as being made by clinicians or patients respectively.
Min-Da Zhao, Xuheng Han, R. Goel et al.· 0 citations
Allocating scarce intensive care resources requires decisions about both which patients should receive priority and which rules should guide those choices. We investigate how the procedural allocation rule "first come, first served" (FCFS) shapes triage decisions. Using experimental ICU bed allocation scenarios, we exa...
Iván Aranzales, H. Chan, David Stadelmann et al.· Social Science & Medicine (1...· 0 citations
Prescriptive analytics uses predictions to recommend decisions, but its superiority over the current approach is commonly evaluated on historical data in which only the cost of the implemented decision is observed. When a prescribed decision differs from the historical decision, its actual cost is a counterfactual outc...
Yunxiang Shu, Ran Yan, Feng Tian et al.· Mathematics· 0 citations
The advent of automated digital health platforms has revolutionized the initial stages of patient care, primarily through the deployment of symptom triage systems. However, a pervasive challenge in these automated environments is the prevalence of missing patient evidence, as individuals often fail to report complete o...
Catherine Turner· International Journal of Com...· 0 citations
Emergency department (ED) triage is a high-stakes prioritization task in which demographic, socioeconomic, and system-context information may improperly influence acuity assignment. Although open-source large language models (LLMs) are increasingly considered for local and privacy-preserving clinical decision support,...
Manar Aljohani, Brandon Ho, Kenneth W. McKinley et al.· 0 citations
Large language models are entering decisions in triage and lending, where task-relevant inference must be distinguished from impermissible proxy use, and causal proxy effects in four LLMs on a clinical-ranking task with known ground truth are measured.
Zeng-Qing Wu, Chuan Xiao· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.