Measurement Invariance as an Item-Screening Criterion for Large Language Model Benchmarks: A Position Note

Abstract

Item response theory is increasingly applied to large language model benchmarks. Recent work uses item response models to select small informative subsets of items, to adapt item administration to model ability, to flag likely label errors, and to estimate latent ability instead of relying on raw accuracy. Recent work also establishes that leaderboard orderings move when the item set changes, whether the items are reordered, replaced by an ability estimate, or removed after an audit. Each of the established uses screens items on a criterion estimated from responses to a fixed presentation of each item. This note argues for a fourth criterion, estimated across presentations instead of within one: an item is a candidate for removal when its response function is not invariant across a change to the test that carries no construct-relevant content. The note maps the configural, metric and scalar invariance hierarchy onto the parameters of a two-parameter logistic model, separates the test-level hierarchy from the item-level removal rule, specifies the dependence structure implied by testing the same models under several presentations, states four hypotheses, and states what would falsify the argument. To the author's knowledge, no prior study has tested a configural, metric and scalar item-parameter hierarchy across controlled presentations of the same large language models and used item-level failure as an exclusion criterion.