Auditing sex/gender disparities in emergency triage with LLM-based paired comparisons

Abstract

We present a domain-agnostic paired-comparison approach that uses Large Language Models (LLMs) to quantify sex/gender-related asymmetries in documented clinical decision-making. The method trains an LLM to emulate observed decisions, then evaluates sex-swapped pairs in which only sex is flipped, holding documented clinical content constant. We apply it to emergency triage, analyzing more than 140,000 Bordeaux University Hospital (France) admissions and testing methodological portability on MIMIC-IV, spanning a different language, population, and healthcare system. Fine-tuning Mistral NeMo 12B for triage prediction and using Mistral Small 24B for pair generation, we find otherwise identical presentations were more likely to receive a lower-severity predicted score as female than male: 1.1% (95% CI 0.9-1.3) in the French cohort, 2.2% (1.7-2.7) in MIMIC-IV. Predictions are sensitive to both tabular and textual sex markers, with the asymmetry emerging primarily in the combined bimodal setting. A model retrained on sex-neutralized inputs eliminated the between-sex prediction gap, indicating the asymmetry is mediated by explicit sex markers. Patterns vary with nurse-patient sex concordance, suggesting the model captures stable features of the recorded data rather than random artifacts. These effects are small and documentation-level. We therefore present this as a methodological feasibility study: LLMs can serve as scalable probes of documented decisions, generating hypotheses rather than establishing bedside clinician behavior or clinically meaningful undertriage, which would require clinician-anchored validation. Beyond emergency care, the approach supports bias audits in other domains.