Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
Inspect_permute is introduced, an open-source extension to the inspect_ai evaluation framework that runs exhaustive answer-order permutations per question and reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals and brackets the detectable region of position-bias measurement.