Jun 2025· EUROMICRO Conference on Software Engineering and Advanced Applications· pp. 143-158· 10 citations· ⚡ 1 influential· 36 references
Computer Science
TL;DR
Five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics are presented, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.
Abstract
Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.
Evaluations (Evals) are a deployment bottleneck for real-world AI applications: public benchmarks rarely match a team's users, context, or policies, and human review is often tedious to scale. Motivated by our work with AI applications in the public sector, this project addresses recurring evaluation challenges encountered when applications must satisfy local policy and governance requirements. We present Kaleidoscope, an integrated workflow for contextual functional evaluation that links persona-based test generation, contextualized rubrics, and human review for reliability-gated automated scoring. Generated test cases are scored against application-specific rubrics; human annotations provide reviewable labels; and LLM judges automate scoring only when their agreement with those labels meets a configured threshold. Kaleidoscope is therefore a practical, inspectable, iterative workflow for product teams. We report early evidence from a three-week pilot across four organizational use cases and custom-rubric judge experiments on 108 annotated Q\&A pairs spanning four domains and 14 evaluation dimensions. The results highlight useful features for end-to-end reliable, automated scoring.
Leanne Tan, R. Jaggi, Shaun Khoo et al.· arXiv.org· 0 citations
Large Language Models (LLMs) combined with Retrieval-Augmented Generation (RAG) have emerged as a practical approach for building knowledge-grounded conversational systems. As these systems are increasingly deployed in data-centric environments, the need for systems that support multi-domain adaptability becomes critical. However, deploying RAG systems across heterogeneous domains remains challenging, as retrieval performance and response quality depend heavily on the choice of embedding models, retrieval strategies, vector stores and language model backends. Existing RAG toolkits either focus on algorithmic benchmarking with limited support for stateful, multi-user deployment or on application development with limited support for component-level experimentation and observability, forcing practitioners to choose between research flexibility and production readiness. We propose a deployment-oriented, modular RAG architecture that enables controlled experimentation across domains while incorporating stateful system services. Pipeline components are independently configurable and extensible as the RAG ecosystem evolves. A dedicated system services layer provides persistent storage, session-aware interaction and role-based access control. In addition, integrated latency instrumentation enables fine-grained performance profiling alongside quality evaluation. We implement and evaluate the system across a Sanskrit philosophical corpus and a biomedical abstract corpus to investigate RAG behavior across heterogeneous domains. Experimental results show that identical configurations can exhibit substantially different retrieval and generation characteristics across domains. Multi-turn evaluation demonstrates that session-aware interaction preserves conversational relevance across turns, while latency profiling reveals non-obvious performance interactions between pipeline stages. Collectively, these findings highlight the importance of modularity, statefulness and observability in the design of RAG systems intended for both deployment and cross-domain research.
Sruthi Satyavarapu, Pramod Gupta, Mohammed Fahad· International Conference on...· 0 citations
Large Language Models (LLMs) are increasingly used in AI-assisted engineering education, enterprise learning, technical knowledge support, and software-development workflows. However, static retrieval-augmented generation (RAG) pipelines retrieve external context for every query, which can increase latency, token cost, and irrelevant-context exposure, while direct generation can produce unsupported or stale answers for policy-sensitive, documentation-dependent, or multisource questions. This paper presents a production-oriented adaptive retrieval framework that treats retrieval as a conditional reliability decision rather than a default execution step. The framework combines query embeddings, similarity-based query memory, feedback-confirmed routing labels, comparison/fallback handling, and observability logs to decide whether a query should be answered directly, routed through retrieval, or escalated for evidence comparison. The system was evaluated in a production-similar controlled test bed using a 1,300-scenario validation workload and public benchmark validation using HotpotQA and RAGTruth. In the 1,300-scenario evaluation, the proposed method achieved 91.31% routing accuracy, 85.00% grounded-answer rate, 85.38% unsupported-answer detection, and 79.85% fallback correctness, while reducing mean latency compared with always-on Static RAG. Paired statistical testing showed significant improvements over LLM-only, Static RAG, and Threshold RAG baselines for routing, grounding, unsupported-answer detection, fallback behavior, and latency. The paper also discusses deployment considerations including query-memory growth, stale no-retrieval labels, knowledge-base version tracking, privacy-preserving redaction, and instructor oversight for responsible educational use.
Ramesh Venkatachalapathy· 2026 IEEE International Conf...· 0 citations
Recent developments in Large Language Models (LLMs) have created new opportunities to automate educational assessment and reduce workload for instructors. However, concerns regarding grading consistency, transparency, and pedagogical reliability continue to limit the adoption of fully automated assessment systems. In this study, we propose a Human-in-the-Loop framework for the automated evaluation of engineering laboratory reports based on Retrieval-Augmented Generation (RAG). The proposed framework integrates text extraction, structured content extraction, contextual retrieval from grading rubrics and laboratory resources, rubric-based evaluation, automatic feedback generation, and instructor validation into a unified grading workflow. The RAG module retrieves contextual information so that the language model can generate assessments that align with the course goals and are based on educational information about the specific assignments, thus ensuring consistent evaluation based on the rubric. The system was tested using a set of 56 laboratory reports collected from undergraduate courses in Electrical and Electronics Engineering. The experimental results indicate a strong agreement between the grades provided by the AI and the instructor, with a Pearson Correlation Coefficient of 0.988, a Mean Absolute Error (MAE) of 3.55, and a Root Mean Squared Error (RMSE) of 3.77. Besides, 89.29% of the reports were scored within ±5 points of the instructor scores. The grading time was reduced from 392 minutes to 84 minutes, a workload reduction of 78.57%. The results demonstrate that the integration of Retrieval-Augmented Generation, rubric-based evaluation, and Human-in-the-Loop validation constitutes an effective approach for AI-supported assessment of engineering laboratory reports, maintaining instructor oversight and educational integrity.
Amina Abbi, M. Skouri, M. Raoufi· International Journal of Adv...· 0 citations
Recent advances in retrieval-augmented generation (RAG) and large language models (LLMs) enable researchers to integrate AI into scientific workflows. However, using proprietary commercial AI systems raises concerns about transparency, reproducibility and privacy, which are essential for scientific practices. To this end, AquiLLM was developed as an open-source modular RAG-LLM framework using open-weight models, designed to support research groups in capturing tacit knowledge. In this work, we present a series of architectural improvements and feature enhancements to AquiLLM, including local embedding and reranking, multimodal capabilities, OpenAI-compatible inference interfaces, user interface improvements, semantic and episodic memory capabilities, and skills support. These enhancements were informed by discussions with domain experts, including astrophysicists and environmental researchers, and represent a step toward AI systems more closely aligned with scientific research practices.
J. Stark, S. Saikrishnan, Vikram Seenivasan et al.· 0 citations
The proliferation of large language models (LLMs) across enterprise, research, and public-sector applications has created an urgent need for rigorous, multi-dimensional evaluation frameworks. This paper presents a comprehensive comparative analysis of seven state-of-the-art LLMs — GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro and Flash, LLaMA 3 70B, Mistral Large, and Claude 3 Haiku — across eight evaluation dimensions: benchmark accuracy, safety alignment, cost efficiency, inference latency, context handling, deployment flexibility, multilingual capability, and scalability. A weighted Multi-Criteria Decision Analysis (MCDA) framework is applied to produce transparent composite rankings from empirical benchmark data using five standardized benchmarks (MMLU, HumanEval, HellaSwag, GSM8K, MATH). Results indicate that Claude 3.5 Sonnet achieves the highest MCDA composite score (0.801), driven by accuracy (90.4% MMLU, 92.0% HumanEval) and safety alignment (4.9/5). Gemini 1.5 Flash emerges as optimal for cost-sensitive deployments ($0.075/1M tokens; 210 tok/s). The paper analyzes architectural trade-offs between dense transformers and Mixture-of-Experts designs, provides a deployment recommendation matrix, and contributes an extensible, evidence-based decision framework for enterprise AI practitioners.
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
What does it take to trust AI-driven HVAC optimization? Our AI Model Factory combines agents, machine learning, reinforcement learning and deterministic checks in a governed workflow designed for messy, real-world building data. The post We built an AI factory for HVAC control appeared first on GPT-Lab.
MIT News · Artificial Intelligence· news.mit.eduAug 3, 2026
LLMs do not get smarter just by remembering more. EvoLib turns experience into evolving knowledge, taking reusable skills and insights that help models learn and adapt across tasks long after deployment. The post EvoLib: Turning experience into evolving knowledge appeared first on Microsoft Research.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
