Skip to content
Log in
#small language model Open access

Retrieval Granularity as Evidence Design in Small-Model RAG Question Answering: A Diagnostic HotpotQA Study

Weimao Ke Lixia Yang Mengyang Xu
Aug 2026 · Applied Informatics · 0 citations · 13 references

TL;DR

Results align with a diagnostic perspective on chunking: using evidence at a task-appropriate level of granularity can improve grounding, auditability, and answer quality, but the observed patterns should be interpreted within the HotpotQA distractor setting, fixed generator, and tested context budgets.

Abstract

Retrieval-Augmented Generation (RAG) has become a practical approach for question answering over external corpora, particularly when answers should be grounded in source documents rather than generated only from model parameters. While recent large language models can process increasingly long contexts, they do not remove the need for selecting, organizing, and auditing evidence, especially when systems rely on smaller local models for privacy, cost, or deployment constraints. In this paper, we frame retrieval granularity as an evidence-design variable for answer grounding in small-model RAG question answering. After a brief exploratory NewsQA phase that motivates the error categories, the main study uses the HotpotQA distractor validation split with 7405 hard multi-hop questions and sentence-level supporting-fact annotations. With Qwen3-8B as the fixed generator, we compare closed-book, fixed-budget whole-context, retrieved-context, gold-document, and gold-supporting-fact conditions while varying retrieval granularity, retriever type, and context budget. Retrieved context substantially outperforms closed-book answering and the 1024-token fixed-budget whole-context condition but remains below gold-document and gold-supporting-fact upper bounds, indicating that retrieval, generation, and evaluation limitations should be analyzed separately. Sentence-level retrieval under-recovers multi-hop evidence, especially for questions with three or more supporting facts, while paragraph-level and moderate token-level chunks recover substantially more complete evidence. In the full condition matrix, hybrid retrieval with 256-token chunks and no overlap achieves an F1 of 0.6816 with a supporting-fact recall of 0.9609, compared with an F1 of 0.6166 and supporting-fact recall of 0.7801 for BM25 sentence retrieval. Additional ablations show that fixed-budget whole-context performance is strongly affected by truncation, that overlap has little practical effect under the tested 1024-token budget, and that a stronger BGE dense retriever improves the best retrieved-context F1 to 0.7027. These results align with a diagnostic perspective on chunking: using evidence at a task-appropriate level of granularity can improve grounding, auditability, and answer quality, but the observed patterns should be interpreted within the HotpotQA distractor setting, fixed generator, and tested context budgets.

DOI Semantic Scholar
Read PDF

Similar papers

Jun 2026

AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering

AB-RAG (Adaptive Budgeted Retrieval-Augmented Generation), a training-free and backbone-agnostic framework that generates an answer, estimates its confidence from a combination of three signals, and then decides whether to stop or to retrieve more evidence, subject to a fixed retrieval budget is presented.

Ansh Kamthan · 0 citations
Book Open access Jul 2026

Rank-ICL: Ranking-based In-context Learning for Search Result Explanation

Evaluation on WikiSA and ExaRank shows that ranking-based few-shot prompting generally improves over zero-shot prompting and achieves competitive performance against random-shot prompting, indicating that retrieval-based demonstration selection is beneficial but not uniformly superior in all settings.

A. Laksito, Aali Alqarni, Mark Stevenson · 0 citations
Open access Jul 2026

Design and Implementation of a Retrieval-Augmented Generation (RAG) Based Chatbot for Document Question Answering

This paper mainly talks about how Retrieval-Augmented Generation frameworks are making document-based question-answering smarter and more reliable than traditional models, and explores different types of RAG techniques like dense retrieval, vector database optimization, generative language model integration, and hybrid search frameworks.

Manthan Meshram, Kunalgir Ridhanarthi, Darshan Khirekar · 0 citations
Preprint Aug 2026

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

We introduce SciRet, a compute-aware empirical study of retrieval-augmented generation for scientific question answering over CORD-19. Rather than proposing a new model, we evaluate a fixed scientific RAG pipeline across three corpus scales: 1,034 chunks (1K papers), 5,160 chunks (5K papers), and 15,480 chunks (15K papers). The pipeline combines sentence-window chunking, BM25, BGE-M3 dense retrieval, reciprocal rank fusion, optional cross-encoder reranking, and grounded answer generation. Across these settings, hybrid retrieval is more robust than either sparse-only or dense-only retrieval in our setting, reaching Recall@10 of 1.000 at 1K and 15K. In contrast, an MS MARCO-trained cross-encoder reranker reduces precision on the scientific corpus, suggesting that domain mismatch can outweigh the benefits of stronger query-passage interaction. Generation faithfulness measured with RAGAS increases with corpus scale in our setup. Retrieval evaluation uses pseudo-relevance labels derived from the hybrid system, so we treat the results as controlled comparative evidence rather than a benchmark claim. We release code, indexes, and evaluation outputs to support replication and follow-up studies.

Kaysarul Anas Apurba, Mahade Hasan, Rofiqul Alam Shehab et al. · 0 citations

Related blog posts