Jul 2026· Journal of Artificial Intelligence and Data Science· 0 citations
TL;DR
The conclusions that can be drawn from this experiment are that threshold-based filtering of context significantly improves the performance of a RAG model.
Abstract
Retrieval-Augmented Generation (RAG), which combines text generation and document retrieval, is one of the more useful methods for maintaining big language model outputs based on actual, verifiable sources rather than the model's own stored knowledge. However, what is initially extracted from the retrieval stage has a significant impact on how well a RAG system responds to a query. Using the Stanford Question Answering Dataset (SQuAD) as the testbed, this study especially examines how the similarity threshold employed during dense retrieval affects both retrieval quality and, later on in the pipeline, the quality of the final answer. The multi-qa-mpnet-base-dot-v1 embedding model was used to encapsulate context passages and questions, which were then compared using cosine similarity. The answer generator was LLaMA 3.1 8B, which was accessed using the Groq API. Fifty questions selected from the SQuAD validation set were used to test eleven threshold values, ranging from 0.0 to 1.0 in steps of 0.1. The quality of retrieval was evaluated by the measures Recall, Precision, and Mean Reciprocal Rank (MRR). The generated answers were assessed by means of Exact Match (EM) and F1 Measure. The baseline performance (at 0.0 threshold) had F1 at 0.827 and EM at 0.72, while for the 0.3 threshold it became F1 at 0.848 and EM at 0.78. Once past the 0.5 threshold, all metrics showed a sharp drop. Overall, the conclusions that can be drawn from this experiment are that threshold-based filtering of context significantly improves the performance of a RAG model.
The Adaptive Multi-Stage Vector Retrieval (AMSVR) framework is proposed, prioritising weighted, drift-resistant composition over uniform fusion, and offers tailored configurations: AMSVR-Scientific (dense + tuned hybrid) peaks at NDCG@10 = 0.7570 on SciFact, while AMSVR-Full (seven stages) targets broader, noisier corpora where Recall@100 matters most.
Samsudeen Alabi Bankole, Yakub Kayode Saheed· NLP & Big Data· 0 citations
Evaluation on a multi-page technical PDF document shows that the hybrid retrieval and re-ranking stages together raise retrieval precision and reduce irrelevant or unsupported answers compared with retrieval limited to a single method, supporting the use of this approach for reliable, document-grounded question answering.
Vishwa K Dave, K. Pallavi· International Research Journ...· 0 citations
Due to the surge in digital document creation, intelligent systems that can effectively retrieve accurate information from unstructured PDF content have become a necessity. Current keyword-based search techniques lack semantic meaning, while large language models (LLMs), when utilized individually, produce hallucinated results when they are not contextually aware of document content. In this paper, we introduce a novel real-time question answering system for PDF content using a lightweight approach to Retrieval-Augmented Generation (RAG). Our system combines a HuggingFace sentence transformer model for semantic embeddings with a FAISS vector similarity search for efficient retrieval, leveraging a Groq-hosted LLaMA 3.3 70B model for context-grounded answer generation. The system is built using LangChain for orchestration and Streamlit for web-based deployment, and requires no GPU, no fine-tuning, and no pre-indexed corpus. Evaluation across five complementary metrics—Exact Match, Token-level F1, BLEU, ROUGE, and semantic similarity—demonstrates strong alignment between generated and reference answers, with a peak semantic similarity of 0.8709 and a mean Token F1 of 0.4701, confirming the viability of this approach for real-world document intelligence.
Mukesh Lakshmi Sai Medikonda, Kalva Vishnu Teja, K. Greeshma et al.· International Conference Com...· 0 citations
Retrieval-Augmented Generation (RAG) helps language models give more accurate answers by using information from external documents. Instead of depending only on what the model learned during training, it can search for relevant information when needed. The quality of the retrieved information is important for getting good results. This paper reviews different ways to improve retrieval, such as rewriting queries, adding related terms, breaking complex questions into smaller parts, generating a sample answer before searching, and refining queries through multiple steps. A simple example is included to show how these methods can improve search results. The paper also compares the techniques and discusses their advantages and disadvantages. Finally, it highlights some common challenges, including slower response times, difficulties in measuring performance, and the possibility of changing the user's original meaning
Ayush Aryan, Sudhakar Ranjan· International Journal For Mu...· 0 citations
Experimental results show that SAC-RAG reduces token consumption by 38%–58% at the cost of only a 1–2 percentage point EM drop, with EM actually improving after compression for reasoning-type questions, achieving the optimal quality–efficiency trade-off in terms of token consumption.
Deyu Zhang, Hongqiang Yu, Jinze Huo et al.· IEEE Access· 0 citations
The idea of context is no longer considered secondary in the construction of language-model systems. With the use of local Retrieval-Augmented Generation, even a tiny modification of the prompt or the context might produce another set of retrievals, citations, and ultimately different answers; however, in practice, tests are often performed with only one version of the question. In this work, we suggest a local context-engineering framework for exploring perturbation robustness, reproducibility, and budgetfriendly assessment in one unified pipeline. Local RAG is built on several small teaching packs for the models; perturbations are introduced to the queries, each experiment is recorded in capsule format, and a gate mechanism based on the decision tree is used to judge if the whole perturbation suite can be skipped, minimized, or run. As a result, the augmented capsule-derived data set contains 3,570 perturbation rows, out of which 2,619 labeled rows are used for retraining. In this labeled subset, BLEU scores are available in 446 cases, answer perplexity and semanticjudgment scores are available in 1,668 cases, citation overlap is available for all rows, and retrieval overlap is available for 2,099 rows. The final decision tree obtains 0.915 ROC-AUC, 0.891 PR-AUC, 0.913 accuracy, 0.863 F1 score, and expected savings of 76.5% on the held-out augmented test split. A more conservative threshold setting lowers the stable-risk value from 0.084 to 0.062, while reducing expected savings to 57.4%. Thus, we provide a local study pipeline to examine the effect of perturbations on answer stability, contradictions, robustness, and budget-friendly evaluation.
Rahul Reddy Gangapuram, William B. Andreopoulos· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
