The Evidence You Do Not Give the AI: Holdout Asymmetry and Epistemic Self-Correction Under Model Mismatch

Abstract

This work introduces holdout asymmetry: the epistemic asymmetry created when a human evaluator can reserve evidence outside a language model's conversational context and later reveal it after the model has committed to an explanatory position. It develops the associated concepts of prediction vacancy and degenerating accommodation, and proposes the Reserve → Commitment → Reveal → Revision protocol for evaluating whether multi-turn language models genuinely revise explanatory structures under model mismatch or instead preserve them through post-hoc auxiliary explanations. Version 2 adds the operational HOLDOUT-Bench validation package. The release includes the revised benchmark and human scoring manual (HOLDOUT-Bench v0.4), a blinded Gold Calibration Set v0.2, the corresponding Gold Calibration Key v0.2, and the gold-key dispute procedure. These materials operationalise the conceptual framework through synthetic, ground-truth cases in which healthy revision, progressive auxiliary reasoning, degenerating accommodation, prediction vacancy, unsupported mechanism commitment, appropriate abstention, and sensitivity to non-diagnostic evidence can be separately measured. The validation architecture explicitly distinguishes reliability from validity. Human raters must recover intended codes on a synthetic gold set as well as demonstrate inter-rater agreement on an independent validation set. The benchmark also includes mechanisms through which disagreement can expose defects in the item key itself, allowing the evaluation instrument to undergo the same form of epistemic correction that it demands from the systems being evaluated. Version 2 therefore extends the original theoretical contribution into a reproducible benchmark specification intended for empirical evaluation of multi-turn epistemic self-correction in large language models.