Intent-Aware Adaptive Trust Retrieval-Augmented Generation (IAT-RAG): A Framework for Trustworthy Document Question Answering Using Open-Source Large Language Models
Jul 2026· International Journal of Creative and Open Research in Engineering and Management· Vol 02, pp. 1-9· 0 citations
TL;DR
The paper summarizes the evolution of the conversational AI, Transformer-based LLMs, and RAG architectures, provides an illustrative evaluation protocol and literature-based comparison, and summarizes comparative results which suggest that the IAT-RAG system should outperform all seven baselines on the tasks of citation accuracy and retrieval F1-score, particularly on multi-hop and comparative queries.
Abstract
Large Language Models (LLMs) show impressive natural-language understanding and generation skills but are limited to knowledge that is pre-stored and trained, and can hallucinate when asked about private, domain-specific or newly generated documents. While conventional RAG and its recent variants (Hybrid RAG, Corrective RAG (CRAG), Adaptive-RAG, FLARE, RAPTOR) each make improvements to only a single stage of the retrieval pipeline, they do not effectively condition retrieval on the intent of the query and quantify the trustworthiness of evidence before generation. We introduce the Intent-Aware Adaptive Trust Retrieval-Augmented Generation (IAT-RAG) framework that integrates (i) query-intent classification to guide each query to a suitable retrieval strategy, (ii) an Adaptive Trust Score (ATS) to adjust the retrieval confidence based on the model's intent classification and (iii) an Evidence Quality Score (EQS) to filter the credibility and internal consistency of each retrieved passage before it is used for generation. Conventional RAG passes all top-k retrieved passages to the generator, whereas the passages supplied to the generator are only those that meet Combined Trust threshold from an intent perspective. The paper summarizes the evolution of the conversational AI, Transformer-based LLMs, and RAG architectures, provides an illustrative evaluation protocol and literature-based comparison, and summarizes comparative results which suggest that the IAT-RAG system should outperform all seven baselines on the tasks of citation accuracy and retrieval F1-score, particularly on multi-hop and comparative queries. The current work is a design and protocol stage contribution, whereas the actual implementation is fully open-source (Sentence-Transformer embeddings, a FAISS vector index, and an open-source instruction-tuned LLM), while complete empirical validation on real data, including statistical-significance testing, is identified as the next immediate step.
Keywords: Retrieval-Augmented Generation, Large Language Models, Intent Classification, Adaptive Trust Score, Evidence Quality Score, Hallucination Mitigation, Open-Source LLMs.
Large Language Models (LLMs) offer strong capabilities for Natural Language Processing, yet their inherent uncertainty often produces hallucinations, confident but incorrect statements, which is critical in domains requiring precise knowledge representation. Retrieval-Augmented Generation (RAG) reduces this risk through information retrieval, but standard pipelines still suffer from fragmented context and weak alignment between queries and legal provisions, limiting trustworthy knowledge extraction. This study proposes a Metadata-Aware RAG architecture to improve grounding in large legal corpora. It integrates: 1) Sub-chunking with Legal Metadata Inheritance, which transforms unstructured legal PDFs into granular, metadata-rich fragments; and 2) an Adaptive Filter Creator, a pipeline that extracts structured constraints and compiles optimized hybrid retrieval queries. These components enhance semantic alignment, reduce uncertainty-driven hallucinations, and strengthen neural information retrieval. Using a curated Peruvian labor law corpus and 150 manually validated question–answer pairs, the system was evaluated across three LLMs (Llama-3.1-8B, GPT-OSS-20B, Gemma-3-27B). The proposed architecture achieves double-digit improvements over a Naive RAG baseline across all four RAGAS metrics—Context Precision, Context Recall, Factual Correctness, and Faithfulness—with gains ranging from 13.10% to 28.15%; notably, Faithfulness surpasses 0.90 for Gemma-3-27B. Statistical analysis confirms significance (t(11) = 15.49, p = 4.06 × 10−9) with an extremely large effect size (Cohen’s d = 4.47). Regression results show minimal influence of model size (slope < 0.005), indicating that retrieval design has a stronger influence than parameter count in the evaluated setting.
Alexandra V. Jove-Ticona, Luis J. Duarte-Coaquera, Israel N. Chaparro-Cruz et al.· International Journal of Adv...· 0 citations
Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit.
Alden Do Rosario, Hussein Younes, Felipe Pires· 0 citations
SCORE-RAG reformulates multi-hop RAG as a two-phase adaptive process: exploration for dynamic query understanding, followed by exploitation for precise evidence gathering, which enables adaptive query comprehension, reduces error accumulation via self-verification, and produces interpretable reasoning chains for accurate answer generation.
Shuran Zhou, Rui Ling, Junan Chen et al.· Annual International ACM SIG...· 0 citations
The evidence indicates that no single RAG or vector-database configuration dominates across retrieval quality, faithfulness, latency, throughput, storage, cost, and scalability, and the review positions RAG–vector database integration as a joint retrieval-and-systems optimization problem rather than a database-selection problem alone.
Muhammad Fuad Bin Abdullah, Safwan Abd Razak, Noorrezam Yusop et al.· International journal of res...· 0 citations
Automated customer support technologies based on large-scale language models (LLMs) suffer from three classic failure modes: false creation of non-existing policies, inconsistent answers to the same questions, and misleading interpretations of user intentions due to confusing, joking, or vague inquiries. Large-scale language model (LLM)-based automated customer support technologies fail due to false policy creation, inconsistent answers to the same questions, and misleading interpretations of user intentions due to confusing, joking, or vague enquiries. Instant-Guided Retrieval-Augmented Generation with LoRA-based Intent Routing, a novel customer assistance system with intent-based retrieval, was successfully created. The Baseline pre-trained Llama-3-8B-Instruct model with no external grounding, a Naive Retrieval-Augmented Generation pipeline (Solution V1) that embeds SOP passages with sentence-transformers/all-MiniLM-L6-v2, indexes them in ChromaDB, and augments the prompt with the top-retrieved passage at inference time, and a Hybrid, The same externally held test partition was striated and an adversarial partition was screened for hedging and frustration language to evaluate all three procedures. Evaluations used ROUGE-1, ROUGE-L, BLEU, Format adherence rate, exact match, and fuzzy match accuracy. Based on 50 samples, retrieval increased ROUGE-1 from 0.2116 to 0.277, a 31.4% increase over the ungrounded benchmark, and BLEU from 0.0030 to 0.0328, an almost tenfold gain. With a fine-tuned intent router added to retrieval, ROUGE-1 would rise 3.9% to 0.2887 and BLEU to 0.0478 (+45.4% over V1). For the entire and adversarial sets, the intent router adheres to JSON Format 100% with 25-36% exact match and 42-46% fuzzy match accuracy.
Unknown authors· International journal of com...· 0 citations
Retrieval-augmented generation (RAG) improves large language models by grounding generation in external evidence, but it also introduces a source trust problem: retrieved context may be useful, irrelevant, or even misleading. Existing RAG systems often apply a fixed trust policy toward retrieved evidence, which can either over-trust incorrect context or underuse context when the user explicitly asks for context-following behavior. Therefore, we propose Intent-Guided Decoding (IGD), a framework that arbitrates between retrieved context and parametric memory according to user intent. IGD uses answer-level filtering and token-level correction to steer the final decoding trajectory between retrieved context and parametric memory. We evaluate IGD on three faithful QA benchmarks and three factual-conflict benchmarks across five LLMs, IGD substantially improves factual recovery, achieving gains of up to 65.4 percentage points on factual-conflict benchmarks over Direct RAG, while preserving or improving strict context-following behavior, this findings highlight the importance of balancing factuality and faithfulness in RAG.
