Evaluating the Validity of a Spoken Dialog System-Mediated Role-Play Test for Assessing Second Language Oral Communication

Abstract

Generative AI offers new opportunities for interactive oral communication assessment for specific purposes by building on past research investigating test takers’ oral interaction with spoken dialog systems (SDSs). Performing as automated conversational agents, SDSs can address logistical challenges in classroom-based oral assessments where interactive tasks are difficult to implement. This study investigated the validity of interpretations of a prototype SDS-mediated Tourism English Speaking Test (SDS-TEST) developed using evidence-centered design. Guided by the interpretation/use argument framework, this study focused on supporting the generalization and explanation inferences. Thirty Turkish English as a Foreign Language (EFL) students in a Tourism and Hotel Management program completed three SDS tasks, each rated by four trained raters. The generalization inference was supported using a univariate G-study with a fully crossed, two-facet design on 360 composite scores, while a D(ecision)-study informed optimal task and rater configurations for future designs. The explanation inference was supported through correlational analyses with related assessments and qualitative insights from participants’ strategy use gathered via stimulated recalls. Findings support the inferences in the validity argument for the SDS-TEST, highlighting the potential of SDSs in English for Specific Purposes oral assessment and offering a model for exploring validity in the next generation of oral communication assessment.