Interpreting ASES-generated evidence for character assessment in higher education: A critical narrative review

Abstract

Automated short essay scoring was built to reduce the burden of marking constructed responses. Because it scores by computing similarity between texts, it also yields a measure of how closely two student responses resemble one another, and a growing body of work now treats that by-product as evidence about student conduct, and specifically about honesty. This critical narrative review examines whether such evidence can validly support inferences about students’ honesty or broader character. It traces the movement of automated scoring from surface features to semantic representations, compares it with the other technologies institutions use to protect integrity, and examines how honesty has been decomposed into indicator sets in studies that make character claims. It then sets out the inferential chain such claims require: that resemblance arose from copying rather than from a constrained task or a shared source; that the direction of copying is known; that the behavior recurs; and that it belongs to the construct named. The literature supports the first link well and the remaining three sparingly, relying largely on expert agreement about instrument content rather than on evidence that scores mean what they are claimed to mean. We specify the psychometric evidence required to close the gap, and argue that the case is further constrained by time: text generated by a large language model resembles no other submission and is invisible to similarity-based detection, so a system built to find copying between classmates may register the least honest work as the most original. Automated scoring is well evidenced for efficiency, defensible for integrity monitoring when a flag initiates a human process, and not yet warranted as a measure of character.