The Cell, Not the Call: A Preregisterable Audit Protocol for Repeated-Call Behavioral Experiments With Large Language Models

Abstract

Abstract. When a hosted large language model answers the same prompt many times, the resulting completions can estimate that model's response distribution for that one prompt. They cannot, on their own, stand in for a sample of scenarios, participants, or models, and treating them as if they could undermines the claims that psychological researchers most want to make with these systems. This article presents an auditable protocol for closing that gap: decisions, records, and prespecified consequences that a researcher adopts before, during, and after data collection, organized around five modules covering the inferential unit, calibration, response handling, endpoint stability, and provenance. We illustrate the protocol's use, rather than test its overall effectiveness, with a four-model moral-judgment experiment (11,200 main trials) in which it surfaced substantial calibration ceiling effects, provider-specific parsing problems, and a registered stability gate that excluded a model with a maximum observed change of 0.10 while retaining models that changed by 0.55 and 0.45. A supporting simulation, reported in the supplementary material, shows why aggregating repeated calls into cell-level counts does not, by itself, correct for unmodeled dependence: aggregated and call-level analyses are mathematically equivalent, so both inherit the same inflated false-positive rate once cell heterogeneity is ignored. The protocol's contribution is practical rather than statistical. It gives psychological researchers a way to convert dependence-and-validity concerns into decisions, thresholds, and inspectable records that a reviewer or replicator can check, with explicit limits on what one worked case and one focused simulation establish.