Development and validation of the generative AI assessment literacy scale for higher education students: psychometric evidence and associations with feedback engagement and academic integrity

Abstract

Students now use generative artificial intelligence (GenAI) to interpret assignments, revise drafts, check evidence, and respond to feedback on assessed work. However, students’ capacity to use GenAI responsibly within assessment contexts remains insufficiently conceptualized and measured. This study developed and validated the Generative AI Assessment Literacy Scale (GAA-LS) for higher education students. A two-study scale development and validation design was used. An initial pool of 30 items was generated from literature on assessment literacy, AI-supported assessment, feedback engagement, and academic integrity. After expert review and pilot testing, Study 1 conducted item analysis and exploratory factor analysis with 486 students. Study 2 examined confirmatory factor analysis, reliability, convergent and discriminant validity, measurement invariance, criterion-related validity, known-group validity, and structural associations with 798 students. The final 18-item scale supported a five-factor structure: assessment criteria awareness, AI-task appropriateness judgment, verification and evidence checking, ethical attribution and academic integrity, and feedback uptake and revision literacy. In Study 1, exploratory factor analysis indicated good sampling adequacy (KMO = .931) and a five-factor solution explaining 67.82% of the variance. In Study 2, the five-factor CFA model showed acceptable fit, χ 2 (125) = 344.72, CFI = .954, TLI = .944, RMSEA = .047, and SRMR = .042. Internal consistency was satisfactory to strong, with alpha ranging from .83 to .88 across subscales and .93 for the total scale. Evidence of convergent, discriminant, criterion-related, and known-group validity was obtained, and measurement invariance was supported across gender, discipline, and AI-use frequency groups. GAA-LS scores were positively associated with feedback engagement and academic integrity intention, with a significant indirect association through feedback engagement. The GAA-LS provides a psychometrically supported instrument for assessing students’ assessment-specific capacity to use GenAI responsibly. The findings contribute to applied measurement, feedback design, and academic integrity policy in AI-supported higher education.