By evaluating paper retrieval, evidence grounding, and answer accuracy separately, LitTraceQA provides a testbed for scientific QA systems that produce verifiable answers rather than unsupported summaries.
Abstract
Scientific literature is increasingly used as a knowledge source for language models, retrieval-augmented generation systems, and research assistants, but answering research questions from papers requires more than fluent generation. A reliable system must identify the relevant papers, locate the concrete evidence that supports the answer, and produce a response that is faithful to that evidence. We present LitTraceQA, a benchmark for literature-grounded question answering over scientific papers. Given a research question and a metadata pool of papers, a system must return three connected outputs: canonical paper identifiers, supporting evidence locations, and answers in one or more requested formats, including free-form text, multiple-choice answers, and structured tables. LitTraceQA targets evidence types common in scientific reading: tables, figures, text spans, equations or algorithms, and citation contexts. The public development split contains 55 examples, including 26 hidden-source single-paper questions and 29 multi-paper questions, and provides gold papers, evidence annotations, and answers for local validation. We also analyze a larger final annotation collection with 4,978 unique-question records over 4,859 unique gold papers. By evaluating paper retrieval, evidence grounding, and answer accuracy separately, LitTraceQA provides a testbed for scientific QA systems that produce verifiable answers rather than unsupported summaries.
Results align with a diagnostic perspective on chunking: using evidence at a task-appropriate level of granularity can improve grounding, auditability, and answer quality, but the observed patterns should be interpreted within the HotpotQA distractor setting, fixed generator, and tested context budgets.
Large language models are increasingly used to assist scientific reading, but existing evaluation methods often fail to detect whether answers are supported by verifiable citations. We introduce ResearchQA, a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers spanning eight domains and four question types: lookup, comprehension, multi-hop, and adversarial. ResearchQA is designed for citation-grounded evaluation: it permits multiple valid supporting passages for a claim and rewards grounded refusal when the source paper does not support an answer. We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting using a deterministic citation matcher and an LLM-based rubric evaluator. Citation-based metrics separate systems more clearly than LLM-evaluator scores: section coverage and citation accuracy vary substantially across models, while evaluator scores remain tightly compressed. We further find that open-weight models approach the best closed-model citation accuracy while achieving 3 to 6 times lower per-example latency. We release the benchmark, evaluation harness, and evaluator prompt.
LLM-driven construction can populate a knowledge graph (KG) with large volumes of triples extracted from documents, yet hallucination leaves their quality far from guaranteed. Existing verification methods return only a single label, one that fails to indicate which part of the triple is at fault and offers no evidence to check against. We present RA-QGQA, which recasts triple verification as a question-driven, corpus-grounded task. For each triple, a Question Generation (QG) module poses one question per slot (head, relation, and tail); a Retrieval-Augmented Generation (RAG) Question Answering (QA) module retrieves from the corpus and answers each independently; the answers are then matched against the triple to reach a verdict. The verdict pinpoints the faulty slot and carries a passage-level evidence chain the user can trace back to the source. We demonstrate RA-QGQA as an interactive web system in which users import a KG and its source corpus, verify all triples in a single pass, and, for any flagged triple, inspect the passages that justify its verdict. The system thus turns large-scale KG verification into a scalable, evidence-backed process.
Siyang Liu, Hong Duc Nguyen, Yunmiao Li et al.· Proceedings of the 2026 ACM...· 0 citations
Technology scouting and competitive intelligence increasingly rely on rapid access to scientific and technical evidence. Large language models (LLMs) are reshaping question answering by enabling users to express exploratory and evolving information needs in natural language. However, their usefulness is limited by hallucinations, weak calibration, poor abstention when evidence is missing, and the cost of keeping knowledge up to date. In such settings, the key question is whether LLM-generated answers can be trusted for decision support, that is, whether they are faithful to available evidence, transparent about uncertainty, and easy to verify. Recent work shows that hallucinations remain pervasive, while retrieval-augmented generation (RAG) has become the main strategy for grounding answers in external evidence. Yet even strong RAG systems often cite unsupported claims or generate plausible but fabricated references, especially under poor retrieval conditions. In response, prior work has examined citation faithfulness, decomposed generations into checkable units, and proposed factuality verification models and benchmarks. Still, the literature underexplores how to filter and validate evidence when sources are sparse, conflicting, or rapidly changing, as is common in technical and academic search. This thesis addresses that gap by studying faithful LLM-assisted question answering for technical and academic search as an end-to-end problem spanning collection construction, model adaptation, grounded generation, evidence verification, and user-centered evaluation. It is organized around five questions: how LLMs can support exploratory search beyond keywords; how retrieval and generation pipelines can produce trustworthy responses; how systems should behave when evidence is sparse or unreliable; how usefulness and trustworthiness should be evaluated; and which architectures and optimization strategies enable reliable real-world deployment. The thesis argues that truthful technical search requires explicit mechanisms for claim-level grounding, calibrated abstention, contradiction handling, and support-aware evidence presentation. Methodologically, the work follows a pipeline perspective in which datasets, models, and evaluation procedures co-evolve. The empirical setup combines public QA and RAG benchmarks with technical and scientific collections tailored to technology scouting scenarios. A central methodological contribution will be the design of scenario-based evaluation subsets targeting null-retrieval, conflicting evidence, noisy or weak-signal evidence, and exploratory information needs. On top of these resources, the thesis will compare single-pass RAG and more structured agentic pipelines while varying retrieval strategies, prompting policies, claim-evidence alignment procedures, quotation and citation behaviors, and model adaptation techniques such as instruction tuning, parameter-efficient fine-tuning, and reinforcement-based optimization. Evaluation will be multi-dimensional. In addition to answer utility and retrieval effectiveness, the thesis will measure citation faithfulness, support coverage, hallucination and false citation rates, abstention quality, uncertainty communication, and robustness under perturbed retrieval conditions. Whenever feasible, these automatic analyses will be complemented with analyst-oriented user studies that examine whether different evidence presentation formats help users reduce verification cost in realistic scouting workflows. Overall, the expected contribution is a principled and deployable framework for trustworthy natural-language access to scientific and technical information, combining faithful generation, verifiable evidence selection, and evaluation protocols tailored to real technical and academic search.
Juan Luis Filgueiras· Annual International ACM SIG...· 0 citations
Key personnel turnover creates knowledge gaps in document-based service organizations, where information is distributed across technical specifications, operational databases, and team discussions. This study develops a multi-source Retrieval-Augmented Generation (RAG) based Question Answering (QA) system that automatically integrates heterogeneous knowledge sources through a unified source parameter. Using the Adapter Pattern, the system converts PDF/TXT documents and PostgreSQL tables into a common representation, builds a FAISS vector index, retrieves relevant context, and generates grounded answers with Gemini 2.5 Flash. Evaluation employs eight metrics and three composite scores: Knowledge Transfer Effectiveness (KTE), Multi-Source Retrieval Score (MSRS), and Answer Quality Index (AQI). Experiments were conducted on the BOND_SYS dataset using 25 Indonesian questions covering specification documents, an 8-table PostgreSQL database, and 908 developer discussion messages. Results show perfect retrieval performance (Precision@K = 1.000; MRR = 1.000) across all scenarios. The full hybrid configuration achieves the highest Overall score (0.373), while Scenario C records the highest MSRS (0.825). Scenario E obtains ROUGE-L = 0.181 and BLEU-1 = 0.196 using five manually curated reference answers. Two baseline comparisons further support this contribution: a zero-shot LLM without retrieval correctly answered only 8% of questions, while a BM25 keyword-search baseline, competitive on single-source scenarios, was outperformed on cross-referencing tasks, underscoring the added value of dense multi-source retrieval. The findings demonstrate that integrating formal documents, structured databases, and discussion logs enhances knowledge transfer and question answering for organizational support and employee onboarding.
Krisna Dwi Setya Adi, Ivan Michael Siregar· Jurnal Ragam Pengabdian· 0 citations
Scientific papers contain fine-grained records of problem solving: authors mention technical obstacles and methods that were used to address them, often along with reasoning on why those methods were chosen. We introduce MUSE (Mining Underlying Scientific Explanations), a full-text, multi-domain resource of scientific Problem-Solution-Rationale (P-S-R) triplets. We curate 579 expert-annotated full-text paragraphs, with a rich annotation schema covering salient problem, solution, and rationale spans, solves and rationale_of links and conceptual coreference. A modular extraction pipeline scales this annotation to build a high-quality knowledge base of 37K source-grounded P-S-R triplets. We evaluate the extraction components and include a preliminary experiment training a rationale-supervised LLM for scientific problem solving. Interestingly, we find that rationale supervision improves performance on complex, multi-constraint problems but can harm performance on simpler ones.