Evaluating the Automated Scoring and Feedback provided by the Virtual Writing Tutor on

Abstract

This study explores the nature of automated scoring and feedback provided by an artificial intelligence-powered tool, the Virtual Writing Tutor. It also assessed the quality of the feedback specifically for non-native writers of IELTS Writing Task 1 essays. The study therefore contributes to the general research literature devoted to evaluating writing aids, which as yet has paid scant attention to this particular tool, and relatively little to the actual quality of feedback on writing for specific tasks. Eight students participated in this convergent mixed-method study, conducted during a 30-hour IELTS preparation program. A quantitative student-based perspective came from a 15 item, 5-point Likert scale response, questionnaire administered to students concerning their subjective evaluation of the Tutor’s feedback. A qualitative expert perspective came from researcher review of the automated scoring and feedback provided on the participants’ IELTS Writing Task 1 practice essays. The findings showed that the Tutor provided feedback relevant to all four of the IELTS writing assessment criteria, albeit the feedback was not presented straightforwardly in the IELTS categories. The software often used indicators, rather than more direct measures, and so could be regarded as overlooking important details that a human evaluator would notice. Additionally, the Tutor provided wrong information in some specific lexical and grammatical correction feedback. Students however judged that the Tutor was easy to use and (surprisingly) significantly useful in helping them improve grammatical accuracy, but they did not find it definitely effective in terms of improving spelling, content, vocabulary, and variety of sentence structures.