The Prompt Is a Confounder: A Counterfactual Audit of Demographic Bias in the Text Channel of Medical Vision-Language Models

Abstract

Parts I and II of this series audited and attempted to remediate subgroup disparities in chest radiograph(CXR) classifi ers, and Part II declared an explicit limitation: multimodal systems combining imageswith clinical text were out of scope, because “text-derived features may carry demographic informationmore directly than images.” This paper closes that gap. We audit BiomedCLIP, an open-weightbiomedical vision-language model, on 25,596 radiographs from 2,797 patients in the offi cial NIHChestX-ray14 test split, and ask a question that observational subgroup audits structurally cannot: whathappens to the diagnosis when the patient’s demographics are stated in thepromptwhile the image, themodel, the label, and the decision threshold are all held fi xed?The answer is that the prompt is a diagnostic input. Naming a demographic group in the text changesthe false-negative rate by 15.5 percentage points on average and by up to 73.8 points in the worst cell,fl ips the binary call on a median 14.1% of positive cases, and degrades AUC by up to 0.137 — the last ofwhich matters because a threshold cannot change AUC, so that component is not an operating-pointartifact and no post-hoc correction can absorb it. Every one of these fi gures is reported as excess over abank of content-free qualifi ers (“a hospital patient”, “a patient referred for imaging”), which weintroduce as a necessary control: the format eff ect alone produces an apparent gap of 8.9 points,comparable to race’s 12.0, so an audit lacking this control would attribute most of a grammatical artifactto demography.We replicate on two further models spanning a domain-specifi city axis — PubMedCLIP (radiologycaptions) and OpenAI CLIP (general web) — and the replication both strengthens and corrects theaccount. The eff ect appears in all three, in 365 of 375 cells atq< 0.05, and it islargest in OpenAI CLIP,which detects no fi nding above chance: 28.9 points of excess FNR and a 30.3% fl ip rate from a modelwith no radiographic competence. Prompt-channel bias is therefore not a model applying clinicaldemographic priors; it is a property of contrastive image–text pretraining with a pair readout. Againstthat, the ordering across descriptor families doesnotgeneralise — socioeconomic descriptors areBiomedCLIP’s second-largest family and PubMedCLIP’s smallest — so we report that as BiomedCLIP-specifi c rather than as a property of medical VLMs. Age descriptors dominate in all three.Three results explain and constrain the eff ect. First, for the standard positive/negative prompt-pairreadout the perturbation isexactly rank one— verifi ed to 4.6 × 10⁻⁶ across 420 (fi nding × descriptor)cells in every one of the three models — so it is a single fi xed direction independent of the image andtherefore not indexed by the patient’s true group. This places prompt-channel biasupstreamof everydecision-rule remedy in Part II’s stage taxonomy: group-specifi c thresholds provably cannot remove it.Second, the eff ect decomposes into an image-independent component that behaves like an uncontrolledthreshold off set and an image-specifi c component that re-ranks patients; the latter is 26–43% of themean-square shift and is irreducible. Third, because the design is paired at the image level, itsminimum detectable eff ect is 3.4 points against 9.4 for an equivalent observational audit — anobservational study would need roughly 5.9× more positive cases — which dissolves, for this class ofbias, the audit-power obstacle Part II quantifi ed.A positive control validates the congruence null. Section 7 fi nds that the model responds to a stated sexbut essentially not to whether it is true, which is only meaningful if the estimator can detect evidenceuse at all. Substituting view position — recorded in the metadata and plainly visible in the radiograph— yields a diff erence-in-diff erences 16.9× larger, signifi cant in 11 of 11 fi ndings against 1 of 11 for sex, atgreater precision. The sex null is substantive, not a power failure. For mitigation we compare prompt symmetrisation, which inserts the descriptor into both prompts ofthe pair, against the orthogonal and calibrated text-side projections of Chuang et al. On BiomedCLIPsymmetrisation reduces mean absolute excess FNR from 15.5 to 5.2 points at no utility cost, while bothprojections reach only 8.4–9.5 points and cost 6–7 AUC points. We are explicit that symmetrisation isthe zero-cost degenerate limit of Chuang et al.’s calibration objective rather than a new idea. It also doesnot always work: it reduces the eff ect by about 60% on BiomedCLIP and OpenAI CLIP but is inert onPubMedCLIP. We proposed, and Section 10.4 withdraws, a text-only statistic intended to predict thatfailure in advance: it is contradicted by OpenAI CLIP within these same results, and by twohistopathology encoders out of domain. Whether symmetrisation will work must therefore bemeasured on the model in question, which is cheap but not free. We conclude that any deploymenttemplating patient metadata into a promptable diagnostic model has introduced a bias channel that itsimage-side audit cannot see and its threshold policy cannot fi x.