A Lightweight LINE-Based Knowledge Chatbot for Mobile Technical Certification Learning: An Empirical Study with ChatGPT as a Baseline

Abstract

This study developed a lightweight LINE-based mobile knowledge chatbot for the Level B Computer Hardware Repair certification subject and examined its instructional effectiveness in comparison with ChatGPT. The system was deployed through the LINE mobile messaging platform, allowing learners to access certification-related explanations on their own smartphones through natural-language interaction. Unlike large language model (LLM)-based systems that rely on retrieval-augmented generation (RAG), vector embedding, and index construction, the proposed chatbot was built around a Knowledge Base Recommendation Algorithm (KBRA), which generates instructional responses through structured keyword matching and fuzzy search over a domain-specific knowledge base. This mobile deployment was designed to provide a low-cost, easily accessible, and interactive learning support tool for bounded technical certification domains. A nonequivalent pretest–posttest quasi-experimental design was adopted. The experimental group used the LINE-based KBRA chatbot for mobile self-directed learning, whereas the control group used ChatGPT. Both groups participated in an eight-week intervention and completed pre-test and post-test assessments consisting of 10 open-ended questions. The results showed that both groups achieved substantial learning gains. After controlling for pre-test scores, analysis of covariance (ANCOVA) revealed no significant difference in post-test performance between the two groups, and the effect size associated with group membership was negligible. In addition, the two one-sided tests (TOST) equivalence analyses indicated that the learning gains of the two groups were practically equivalent within a ±10-point margin. These findings suggest that, in this bounded certification-learning context, the LINE-based KBRA chatbot may provide learning support comparable to ChatGPT under the conditions examined. However, this comparability should be interpreted cautiously because the study was conducted with a small sample, a single certification domain, and a specific eight-week self-directed learning setting. The proposed chatbot should therefore be regarded as a feasible and deployment-friendly learning support option for structured technical certification learning, rather than as a general alternative to ChatGPT.