Assessing English Phonological Accuracy Across LLM-Based Chatbots

Abstract

English is the most widely spoken language of the time. Thus, learners, regardless of language backgrounds, increasingly seek at least basic proficiency in it. Since spoken English is its most common form of use, understanding its phonological features and processes proves indispensable. With the growing use of machines for linguistic exercises, large language model (LLM)-based chatbots have become popular tools for language learning. However, their phonological accuracy has received limited scholarly attention. This study, therefore, investigates the accuracy of selected LLM-based chatbots across specific phonological features and examines systematic patterns of errors in their outputs. Using a descriptive qualitative research method, grounded in Chomsky and Halle’s (1968) Generative Phonology, a total of 71 responses from each of six notable LLM-based chatbots were purposively sampled and thematically analyzed. Findings reveal that the chatbots demonstrate high competence in English phonology but lack complete reliability. Given the chatbots' dissimilar levels of competence, Claude achieves the best overall performance, while Copilot records the lowest. Although the sampled chatbots generate outputs that apply underlying phonological rules, their inaccuracies indicate limitations in consistently representing abstract phonological structures. The researcher thus recommends cautious use of chatbots’ phonological outputs and regular improvement of training data by developers. In all, the study provides insights for users of the sampled chatbots learning English phonology and for developers seeking to improve the phonological accuracy and reliability of these systems.