An Audit of Measurement Quality and Answer Bias in a Large Classroom-Poll Corpus

Abstract

Real-time classroom polls are widely used and increasingly generated with automated assistance, yet the questions themselves are rarely evaluated as measurements. We audit a large corpus of authentic classroom polls, 604 items across 47 sessions answered 340,668 times by 2,807 learners, as a measurement instrument. For the 539 items whose correct answer could be established and verified from the lecture transcript, we place every item and every student on a common scale using item response theory and analyse the answer structure of the True/False items. Two findings emerge. First, the polls form a coherent but easy scale of moderate precision (marginal reliability about 0.60), on which roughly a quarter of items barely separate stronger from weaker students. Second, students show a robust tendency to answer True, present at the individual level (77% of students lean True), which meets a milder tendency for items to be keyed False; as a result answer direction predicts difficulty, False-keyed items being about thirteen points harder, and the effect survives controls for item content and for selective answering. Both findings rest on signals a polling system already records, so the same checks can be run as items are generated, before they reach students.