Retrieval-Augmented Generation for Question Answering over a Research Paper: A Hybrid Dense-Keyword Retrieval Study on the Transformer Architecture Paper

Abstract

Retrieval-Augmented Generation (RAG) combines information retrieval with language generation to improve the grounding of generated answers in a source document, addressing a central weakness of purely parametric large language models: the tendency to produce fluent but unsupported statements. This study designs, implements, and evaluates a document-level RAG pipeline using the research paper “Attention Is All You Need” (Vaswani et al., 2017) as the knowledge source. The system extracts the paper's text, forms 44 sentence-based chunks with a target maximum length of approximately 1000 characters, embeds each chunk using the all-MiniLM-L6-v2 sentence-transformer, indexes the resulting 384-dimensional vectors with a FAISS IndexFlatL2 index, and retrieves relevant passages using a hybrid score that combines semantic similarity (weighted 0.7) with keyword overlap (weighted 0.3). The five highest-scoring chunks are supplied as context to Qwen2.5-0.5B-Instruct for answer generation under a deterministic decoding configuration. On a five-question evaluation set covering the source paper's motivation, positional encoding, multi-head attention, parallelization, and main contributions, dense retrieval alone achieved Recall@1 / Recall@3 / Recall@5 of 0.40 / 1.00 / 1.00, while the hybrid retriever achieved a perfect 1.00 / 1.00 / 1.00, indicating that a lightweight lexical signal was sufficient to correct the ranking errors that dense retrieval made at the top position. Manual evaluation of generated answers on a 1-5 scale showed that RAG improved every quality dimension relative to a no-retrieval baseline using the same generator: correctness rose from 2.20 to 3.40 (+54.5%), faithfulness from 2.00 to 3.20 (+60.0%), relevance from 3.40 to 3.80 (+11.8%), and overall quality from 2.53 to 3.47 (+37.2%). Despite these gains, qualitative analysis showed that some generated answers still contained details not explicitly present in the retrieved context, demonstrating that successful retrieval does not, by itself, eliminate hallucination in a small instruction-tuned generator. Because the evaluation uses a single document, five questions, and manual scoring, the reported numbers should be read as a controlled, preliminary case study rather than a general benchmark of RAG performance; the paper discusses this scope explicitly and outlines a concrete path to a larger-scale evaluation. Keywords: Retrieval-Augmented Generation; RAG; Dense Retrieval; Hybrid Retrieval; FAISS; Sentence Embeddings; Question Answering; Hallucination; Transformer Architecture; Small Language Models