Pegi1727/LICA-LLM-Benchmark: Preliminary Psychometric Calibration of Large Language Models for Multidimensional L2 Academic Writing Assessment: The LICA Framework

Abstract

This research repository contains the data, scoring matrices, statistical analyses, and supplementary research materials associated with the study "Preliminary Psychometric Calibration of Large Language Models for Multidimensional L2 Academic Writing Assessment: The LICA Framework." The study introduces and preliminarily evaluates the LICA framework, a multidimensional framework for assessing L2 academic writing across four theoretically distinct dimensions: Language (L), Ideas (I), Cohesion (C), and Academic Appropriateness (A). The framework is designed to move beyond single-score automated writing evaluation by examining whether large language models can reproduce multidimensional human benchmark judgments at the criterion level. Three large language models—GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—were evaluated against a Human Gold Standard using an integer rating scale from 1 to 5 across the four LICA dimensions. The calibration dataset consists of five L2 academic writing essays, with the same scoring matrix used consistently across all analyses. The repository supports three research questions: RQ1: To what extent do AI-generated LICA ratings demonstrate ordinal alignment and absolute-score accuracy relative to the Human Gold Standard? RQ2: What degree of pairwise score-level agreement and directional rating bias is observed between each AI model and the Human Gold Standard? RQ3: To what extent do the AI models demonstrate autonomous categorical consensus with one another across the four LICA dimensions? The statistical analysis includes Quadratic Weighted Kappa (QWK), Mean Absolute Error (MAE), Intraclass Correlation Coefficient (ICC), Bland–Altman analysis, paired bias tests, and Fleiss' Kappa. The analytical design explicitly distinguishes human–AI agreement from AI–AI consensus. The findings provide preliminary calibration evidence rather than population-level validation. Within this five-essay calibration sample, GPT-4o showed exact correspondence with the Human Gold Standard across all four LICA dimensions, while Claude 3.5 Sonnet and Gemini 1.5 Pro showed high but dimensionally variable levels of agreement. The results also indicate that Academic Appropriateness may represent a more challenging dimension for autonomous AI scoring than Language or Cohesion. This repository is intended to promote transparency, computational reproducibility, methodological inspection, and further development of multidimensional AI-assisted assessment of L2 academic writing. Keywords LICA framework; L2 academic writing; automated writing assessment; large language models; generative AI; AI-assisted assessment; GPT-4o; Claude 3.5 Sonnet; Gemini 1.5 Pro; human–AI agreement; AI–AI agreement; psychometric calibration; Quadratic Weighted Kappa; ICC; Fleiss' Kappa; academic writing assessment; language assessment; automated essay scoring