Jul 2026· IJOEM: Indonesian Journal of E-learning and Multimedia· 0 citations
TL;DR
Findings indicate that clear prompt instructions can improve the reliability of RAG-based academic chatbot responses for academic information services and show the strongest improvement in faithfulness and context recall.
Abstract
Background: The increasing use of artificial intelligence in higher education has encouraged institutions to develop academic chatbots that provide faster access to official information. However, Retrieval-Augmented Generation (RAG)-based chatbots still require optimization to ensure accurate, relevant, and context-grounded responses.Aims: This study aims to optimize the performance of a RAG-based academic chatbot by applying zero-shot and few-shot prompt engineering strategies.Methods: A comparative experiment was conducted using 45 in-context academic questions for RAGAS-based quantitative evaluation, while 5 out-of-context questions were used as a qualitative robustness check. The system was developed using Python, LangChain, FAISS, OpenAI, and Streamlit, and evaluated using the RAGAS metrics: faithfulness, answer relevancy, context precision, and context recall.Results: The baseline system achieved an average RAGAS score of 0.8421. After prompt engineering was applied, zero-shot prompting achieved 0.8697, while few-shot prompting achieved 0.8565.Conclusion: Zero-shot prompting produced the best overall performance and showed the strongest improvement in faithfulness and context recall. These findings indicate that clear prompt instructions can improve the reliability of RAG-based academic chatbot responses for academic information services.
This research addresses the common challenge of a lack of context in Question and Answer (QA) datasets in digital education, which limits the reasoning potential of Large Language Models (LLMs). To address this, we optimize an automated retrieval-based dataset generation system that systematically enriches QA pairs with relevant pedagogical context from authoritative digital textbooks. This study conducts a comparative analysis of two major text chunking strategies: sentence chunking and recursive chunking. Although these pipelines are designed for general education applications, they are evaluated here through a case study of Indonesian elementary education materials. To ensure the highest reliability, the workflow performance is measured against a ground truth dataset of 978 entries, manually curated and validated by education experts to ensure pedagogical accuracy, and 781 entries from other subjects. Quantitative evaluation using BERTScore shows that recursive chunking achieves a superior F1 score of 0.748 compared to 0.737 for sentence chunking, with peak performance observed on upper elementary school materials (Grades 5 and 6). These findings were corroborated by the final verification phase through User Acceptance Testing (UAT) with an elementary school educator, where recursive chunking achieved a 'Relevant' score of 22 compared to 17 for sentence chunking. A key contribution of this study is the development and validation of a standardized, automated workflow by experts that effectively overcomes the barriers of manual dataset construction for domain-specific tasks, providing a semantically robust foundation for context-aware educational AI.
V. C. Mawardi, Ayu Purwarianti, B. Trilaksono et al.· International Conference on...· 0 citations
This paper proposes RAG-Test, a developer-centric and unified framework that automates test case generation, attribution validation, and productivity estimation and introduces a theoretical productivity gains model, estimating chatbot efficiency improvements over search engines.
Shashank Parmar· Multimedia tools and applica...· 0 citations
The results show that the RAG architecture provides a scalable alternative for creating precise, contextually grounded conversational agents, thereby mitigating some of the main drawbacks of LLMs.
Rabia Shabbir, K. Talpur, Shakeel Ahmad· ICCK Transactions on Machine...· 0 citations
The study successfully validated that the intelligent chatbot efficiently bridges communication barriers, automates repetitive administrative inquiries, and improves service accessibility, confirming that the application is highly usable, practical, and effective as an inclusive, assistive communication tool.
Gil G. Dialogo, Hershey Alburo-Abugadie, Enrico C. Lucero· International journal of com...· 0 citations
HeuristicEdu is presented, a two-phase pipeline that aligns Qwen2.5-7B toward Socratic tutoring via supervised warm-up and Group Relative Policy Optimization (GRPO), and Scaffolding Effectiveness and Conversation Depth are introduced to evaluate outcomes beyond surface fluency.
Xiaokun Wang, Siyu Song, Wentao Liu et al.· arXiv.org· 0 citations
Universities now use Large Language Models (LLMs) to transform their processes for managing student information. The paper introduces an upgraded chatbot system for Narasaraopeta Engineering College (NEC) which extends previous on-premise LLM chatbot research by providing four new functions. The system uses (1) Retrieval-Augmented Generation (RAG) to create citation-based responses through LlamaIndex and ChromaDB, (2) Context Memory which maintains conversation flow during multiple dialogue exchanges, (3) Voice Input through OpenAI Whisper Speech-to-Text (STT) technology, and (4) Multilingual Support which covers English and these seven languages: Hindi, Telugu, Tamil, Kannada, and Malayalam through IndicNLP. The system tested 60 benchmark questions across four academic categories which included regulations and examination policies and fee structures and multilingual queries and achieved 96.7% overall accuracy with sub-second text response times and 1.0–1.4 second voice response times. The system operates entirely on-premise through Docker which safeguards institutional data privacy while eliminating the need for recurring cloud API expenses. The upcoming development will create Emotion-Aware AI, FAQ Auto-Learning, Student Portal Integration, and a Mobile Application.
M. Yaswanth, Kopparapu Sai Amar Durgesh, Mogili Harsha Vardhan et al.· 2026 7th International Confe...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
