Recall Is Not Execution: a Large Language Model and the Revised Geneva Score [Dataset].

Abstract

Study summary A frontier large language model (Claude Opus 4.8, model identifier claude-opus-4-8, Anthropic) was tested against the revised Geneva score for pulmonary embolism using simulated patient profiles. Two tasks were run. Task 1, weight elicitation. The model was asked to assign numeric weights to the components of the revised Geneva score, ten repetitions per prompt, under two independently worded prompts. Task 2, patient classification. The eight objective variables of the revised Geneva score were exhaustively enumerated to generate all 384 possible patient profiles. The model classified every profile as low, intermediate, or high probability of pulmonary embolism, under two wordings that describe identical patients and carry identical Geneva scores. Each wording was run twice, on separate dates, giving four classification runs in total. Analysis 3, internal coherence and negation density. Two further analyses were run on the classification data, without new model queries. The first tests whether the model's classifications satisfy the ordering that any risk rule with non-negative weights must satisfy: a profile carrying strictly more findings must not be assigned a lower probability category. The second tests, within strata of identical revised Geneva score, whether the number of findings stated as absent shifts the model's category. Both are model-against-itself comparisons and do not depend on the revised Geneva score being correct. The script pe_monotonicity.py reproduces both. No human subjects were involved. All patient profiles are synthetic combinations of the score's input variables.