A controlled, deployment-oriented comparison of a lightweight FAQ chatbot and a Retrieval-Augmented Generation chatbot for academic-service question answering supports a cautious hybrid strategy: FAQ for stable repetitive requests and RAG for contextual, document-dependent questions, subject to broader testing with live queries and stronger retrieval baselines.
Abstract
Academic service units need question-answering systems that respond quickly while remaining aligned with institutional regulations. This study provides a controlled, deployment-oriented comparison of a lightweight FAQ chatbot and a Retrieval-Augmented Generation (RAG) chatbot for academic-service question answering. The novelty lies in evaluating both prototypes against the same version-controlled official-document scope, the same 60-question dataset, category-level service cases, paired statistical tests, inter-rater reliability, and RAGAS retrieval-quality metrics. The questions covered registration, KRS/KHS, schedules, tuition payment, thesis or final project, and administrative letters. Three validators assessed accuracy, relevance, response effectiveness, and response time using official documents as the reference standard. RAG achieved 88.33% accuracy, a mean relevance score of 4.48/5, and 86.67% response effectiveness; FAQ achieved 68.33%, 3.46/5, and 65.00%, respectively. Paired tests confirmed significant advantages for RAG in accuracy, relevance, and effectiveness, while FAQ was significantly faster (0.62 s vs 2.41 s). RAGAS evaluation showed context precision of 0.86, context recall of 0.82, faithfulness of 0.89, and answer relevancy of 0.91. The findings support a cautious hybrid strategy: FAQ for stable repetitive requests and RAG for contextual, document-dependent questions, subject to broader testing with live queries and stronger retrieval baselines.
Findings indicate that clear prompt instructions can improve the reliability of RAG-based academic chatbot responses for academic information services and show the strongest improvement in faithfulness and context recall.
Adi Surya Artayasa, Aniek Suryanti Kusuma, Putu Sugiartawan· IJOEM: Indonesian Journal of...· 0 citations
The results show that the RAG architecture provides a scalable alternative for creating precise, contextually grounded conversational agents, thereby mitigating some of the main drawbacks of LLMs.
Rabia Shabbir, K. Talpur, Shakeel Ahmad· ICCK Transactions on Machine...· 0 citations
Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask course specific questions or verify answers against an instructor's lecture. We report a semester-long deployment of VideoPoints platform with a retrieval-augmented chatbot that answers from course lecture materials and returns timestamped citations. The chatbot retrieves only from the active course, uses chapter summaries to guide transcript ranking, and returns clickable timestamped citations. Students used it for quick lookups and exam review. Across 833 messages, 70.5% included citations, none crossed a course boundary, and when no lecture evidence matched, the chatbot usually declined rather than answering. Among the users, citations were the most consistently useful feature, while practice-question generation was the strongest unmet request. We also evaluated the design on the real-world test split of EduVidQA, a public multimodal benchmark for lecture-video question answering. Our design improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval. Together, the results show that effective deployment depends on course isolation, supported citations, and alignment with students'study practices.
Universities now use Large Language Models (LLMs) to transform their processes for managing student information. The paper introduces an upgraded chatbot system for Narasaraopeta Engineering College (NEC) which extends previous on-premise LLM chatbot research by providing four new functions. The system uses (1) Retrieval-Augmented Generation (RAG) to create citation-based responses through LlamaIndex and ChromaDB, (2) Context Memory which maintains conversation flow during multiple dialogue exchanges, (3) Voice Input through OpenAI Whisper Speech-to-Text (STT) technology, and (4) Multilingual Support which covers English and these seven languages: Hindi, Telugu, Tamil, Kannada, and Malayalam through IndicNLP. The system tested 60 benchmark questions across four academic categories which included regulations and examination policies and fee structures and multilingual queries and achieved 96.7% overall accuracy with sub-second text response times and 1.0–1.4 second voice response times. The system operates entirely on-premise through Docker which safeguards institutional data privacy while eliminating the need for recurring cloud API expenses. The upcoming development will create Emotion-Aware AI, FAQ Auto-Learning, Student Portal Integration, and a Mobile Application.
M. Yaswanth, Kopparapu Sai Amar Durgesh, Mogili Harsha Vardhan et al.· 2026 7th International Confe...· 0 citations
A Retrieval-Augmented Generation (RAG)-based chatbot for academic regulation information services using a locally deployed Large Language Model (LLM) and promising feasibility for supporting document-based academic regulation information services is indicated.
Although Conversational AI significantly improves the delivery of public information, Large Language Models (LLMs) frequently face challenges in adhering strictly to Standard Operating Procedures (SOPs) and rarely leverage interaction logs for strategic analysis. To overcome these limitations, this study presents a novel police public service chatbot built on the Laravel framework, integrating the Google Gemini API with Information Retrieval (IR) methodologies and an official SOP database. The system processes user inquiries through text preprocessing, TF-IDF weighting, and Cosine Similarity to extract relevant documents, guaranteeing responses that are both contextually accurate and SOP-compliant. Furthermore, it systematically records conversation logs to analyze public needs and trends. Comprehensive evaluation via Black Box testing, IR metrics, and qualitative reviews demonstrated superior performance. The proposed system achieved an accuracy of 95.42%, outperforming rule-based (82.14%), standard TF-IDF (89.72%), and baseline Gemini (93.41%) approaches. It also received excellent satisfaction ratings from domain experts (4.83/5) and a cohort of 100 citizens (4.73/5). In conclusion, this architecture effectively combines the conversational fluency of LLMs with the strict procedural precision necessary for robust public service administration.
N. Shalehah, M. Efgivia· INTERNATIONAL JOURNAL OF MAT...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
