Students now use generative artificial intelligence (GenAI) to interpret assignments, revise drafts, check evidence, and respond to feedback on assessed work. However, students’ capacity to use GenAI responsibly within assessment contexts remains insufficiently conceptualized and measured. This study developed and validated the Generative AI Assessment Literacy Scale (GAA-LS) for higher education students.
A two-study scale development and validation design was used. An initial pool of 30 items was generated from literature on assessment literacy, AI-supported assessment, feedback engagement, and academic integrity. After expert review and pilot testing, Study 1 conducted item analysis and exploratory factor analysis with 486 students. Study 2 examined confirmatory factor analysis, reliability, convergent and discriminant validity, measurement invariance, criterion-related validity, known-group validity, and structural associations with 798 students.
The final 18-item scale supported a five-factor structure: assessment criteria awareness, AI-task appropriateness judgment, verification and evidence checking, ethical attribution and academic integrity, and feedback uptake and revision literacy. In Study 1, exploratory factor analysis indicated good sampling adequacy (KMO = .931) and a five-factor solution explaining 67.82% of the variance. In Study 2, the five-factor CFA model showed acceptable fit,
χ
2
(125) = 344.72, CFI = .954, TLI = .944, RMSEA = .047, and SRMR = .042. Internal consistency was satisfactory to strong, with alpha ranging from .83 to .88 across subscales and .93 for the total scale. Evidence of convergent, discriminant, criterion-related, and known-group validity was obtained, and measurement invariance was supported across gender, discipline, and AI-use frequency groups. GAA-LS scores were positively associated with feedback engagement and academic integrity intention, with a significant indirect association through feedback engagement.
The GAA-LS provides a psychometrically supported instrument for assessing students’ assessment-specific capacity to use GenAI responsibly. The findings contribute to applied measurement, feedback design, and academic integrity policy in AI-supported higher education.
Jun-Tao Nie, Ze-Yu Zhang, Xiao-Mei Lu et al.· Frontiers in Education· 0 citations
Artificial intelligence (AI) is becoming increasingly relevant to teacher education, yet evidence remains limited on how pre-service teachers’ AI literacy, risk perception, and academic confidence can be assessed within a coherent but multidimensional framework. This study examined the psychometric properties of the AI Literacy, Risk Perception, and Academic Confidence Questionnaire (AIRPAC-Q) among Chinese pre-service teachers. A cross-sectional survey was conducted with 528 participants recruited from teacher education programmes in China. The sample was randomly divided into an exploratory factor analysis (EFA) subsample (n = 258) and a confirmatory factor analysis (CFA) subsample (n = 270). Psychometric evaluation included expert-based content validation, pilot refinement, EFA, CFA, reliability testing, convergent and discriminant validity, concurrent validity, and known-group validity analyses. The final questionnaire retained 14 items across three complementary dimensions: AI literacy, risk perception, and academic confidence. Expert ratings showed acceptable content validity (I-CVI = 0.83–1.00; S-CVI/Ave = 0.94). The hypothesized three-factor model showed an acceptable fit to the data (χ² = 146.32, df = 74, χ²/df = 1.98, CFI = 0.952, TLI = 0.941, RMSEA = 0.060, SRMR = 0.047) and outperformed two-factor and one-factor alternatives. Cronbach’s α values ranged from 0.82 to 0.88, composite reliability values ranged from 0.83 to 0.89, and average variance extracted values ranged from 0.56 to 0.59. Participants with prior AI use experience scored higher on AI literacy and academic confidence but slightly lower on risk perception than those without such experience. These findings support the AIRPAC-Q as a context-specific multidimensional tool for assessing competence, caution, and confidence in AI-supported teacher education.
Findings may help universities identify students who report difficulty controlling AI use, academic worry, delayed bedtime, academic avoidance, or trouble setting limits on AI use and point to health literacy, self-regulation, and time management as candidate resources for future longitudinal and intervention studies.
Zeyu Zhang, Yinlan Zhang, Xiaomei Lu et al.· Frontiers in Public Health· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
