Evaluating User Experience of a Retrieval-Augmented Generation-Based Customer Service Chatbot

Abstract

— As digital services become increasingly widespread, online real-time customer support has become a key way for consumers to seek help. A good service experience can boost customer satisfaction and loyalty, particularly in the financial sector. However, traditional keyword-based chatbots often fail to capture users ’ intent, limiting the scenarios in which they can be used. This study introduces a system based on Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs), aiming to better understand user queries and provide more helpful responses. Through interviews and sentiment analysis, we explored user reactions and what influenced their emotional responses. Findings revealed that traditional bots often triggered frustration and confusion due to rigid responses and poor understanding. In contrast, the RAG-based system demonstrated stronger natural language handling and was perceived as more empathetic. Some users even reported feeling emotionally supported. However, slow response times and difficulty in handling complex queries remained challenges. Based on these insights, we suggest improving response speed, adding voice interaction, and enhancing response reliability to guide future development of intelligent customer support tools.