Aug 2026· International Conference on Data Technologies and Applications· Vol 11, pp. 216· 0 citations· 24 references
TL;DR
This work evaluates a confidence-adaptive graph-enhanced retrieval layer for metallurgical RAG in a controlled synthetic benchmark with explicitly specified generation and evaluation rules and shows how the retrieval rule behaves under controlled conditions.
Abstract
Metallurgical R&D search is difficult for a practical reason: useful evidence is rarely defined by one keyword. A production-support question can depend at the same time on material grade, process route, defect mechanism, property, test method, and numerical conditions. Conventional retrieval-augmented generation (RAG) pipelines largely treat document chunks as independent text and can therefore miss relations that matter for process monitoring, fault diagnosis, and engineering decision support. We evaluate a confidence-adaptive graph-enhanced retrieval layer for metallurgical RAG in a controlled synthetic benchmark with explicitly specified generation and evaluation rules. The benchmark contains 300 generated heterogeneous records derived from a seven-block source distribution and 30 material–process–defect–property archetypes, together with 180 frozen queries: 60 exact, 60 paraphrased, and 60 multi-hop. The main run evaluates robustness to incomplete structured metadata, with 10% missing and 4% erroneous categorical fields. Entity and relation extraction from raw documents is outside the evaluated scope. We compare BM25, TF-IDF, latent semantic analysis, a sparse + dense hybrid, graph-only retrieval, two ablations, and the proposed adaptive hybrid. On the complete query set, the proposed method obtains MRR = 0.992, Precision@5 = 0.980, Recall@10 = 0.859, and nDCG@10 = 0.948. Relative to the sparse + dense hybrid, nDCG@10 increases by 0.186 (24.4%); the paired 95% bootstrap interval is in the range of 0.166–0.206, and the Holm-adjusted Wilcoxon p-value is 2.59 × 10−29. Under severe degradation with 40% missing and 16% erroneous metadata, the adaptive method retains mean nDCG@10 = 0.791, compared with 0.650 for graph-only retrieval and 0.762 for the metadata-independent sparse + dense hybrid. A 5000-run Monte Carlo analysis estimates 6763 chunks and 58.70 MB for indexed vectors plus metadata at a 512-token chunk size and 64-token overlap. The results show how the retrieval rule behaves under controlled conditions; they are not evidence of plant-level effectiveness or of the quality of generated answers. Those questions require external, expert-labeled validation.
This study develops a multi-source Retrieval-Augmented Generation (RAG) based Question Answering (QA) system that automatically integrates heterogeneous knowledge sources through a unified source parameter to enhance knowledge transfer and question answering for organizational support and employee onboarding.
Krisna Dwi Setya Adi, Ivan Michael Siregar· Jurnal Ragam Pengabdian· 0 citations
Retrieval-augmented generation (RAG) systems for scientific literature require evidence-based choices of document segmentation, representation, retrieval, and generation components, particularly when the source collection varies in topical specificity and document structure. This study addresses the lack of an end-to-end, component-level comparison of these choices for soil science question answering. A three-stage evaluation was conducted across general, domain-specific, and geospatial soil science corpora. The corpus combines foundational soil science books, peer-reviewed research articles, European soil monitoring material, and geospatial mapping publications, thereby covering both broad disciplinary concepts and specialized scientific evidence. The study compares four chunking strategies, three embedding models, five retrieval methods, and five large language models. In Experiment 1, semantic chunking with text-embedding-3-large achieved the highest aggregate retrieval scores (recall@1 = 0.824; MRR = 0.819), whereas text-embedding-3-small delivered practically comparable performance at lower cost. In Experiment 2, hybrid reciprocal rank fusion achieved recall@5 values of 0.957, 0.960, and 0.647 for the general, domain-specific, and geospatial corpora, respectively; the cross-encoder reranker showed weaker rank quality on scientific content. In Experiment 3, model responses attained BERTScore values of 0.909–0.927 and faithfulness of at least 0.993; these automated measures indicate low contradiction with retrieved context but do not establish answer completeness or human-perceived correctness. The study provides a reproducible component-level evaluation design, characterizes the effect of corpus specificity on RAG retrieval, and identifies a practical configuration for soil science literature retrieval. Among the models retained for direct aggregate comparison, Llama 3.1 8B offered the most favorable observed balance of answer quality, latency, cost, and model openness.
Karla Topić, Marina Bagić Babac, Vedran Mornar· Information· 0 citations
This research provides a highly accurate, scalable, and reliable framework for automated bridge defect analysis, offering a practical methodology to enhance data utilization in bridge management.
Lu-yang Zhang, Xuzhao Lu, Fengzong Gong et al.· Advances in Structural Engin...· 0 citations
LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost, and is the first to score value accuracy, record completeness at scale, grounding, and measured cost together.
Boyang Zhang, Adrian Lyjak, Elizabeth Stewart et al.· arXiv.org· 1 citation
Results indicate that integrating multi-source domain knowledge with relation-preserved retrieval and attribute-supported filtering provides more focused and inspectable evidence, thereby supporting more accurate complex material question answering.
Peize Li, Xi Guo, Nan Yin et al.· Electronics· 0 citations
Scientific papers in materials science contain critical experimental details (e.g., reagents, synthesis conditions, and measured properties) for building structured databases and enabling downstream analysis, but large-scale structured extraction remains difficult. Classic rule-based systems rely on hand-written patterns and pre-defined targets, which can be brittle when high-value fields are domain-specific or shift across subdomains. Recent LLM-based extraction is more flexible, but it often still depends on manual schema and prompt design, where a schema specifies the target fields to extract into a table. We present AutoSchema, an iterative literature-mining pipeline that bootstraps both retrieval and schema induction from a small set of seed papers, then freezes the induced schema for scalable, evidence-grounded extraction. AutoSchema is not fixed-field extraction with a manually specified field list (e.g., ChatExtract-style prompting), not entity/ontology-style recognition and normalization (e.g., MatScholar), and not rule-based pattern engineering. Instead, it targets scalable literature mining under schema uncertainty, where the schema must be induced, stabilized, reused, and audited. Starting from 3–5 seed PDFs, AutoSchema expands a domain corpus via multi-source retrieval, induces and refines a compact schema set across rounds, and performs schema-conditioned, evidence-grounded extraction with JSONL audit logs and flattened tables. Across five independent seed sets, lightweight canonicalization and semantic normalization substantially improve schema agreement (mean pairwise Jaccard similarity from 19.97% to 43.41%; field-level overlap from 32.50% to 59.91%). In a covalent organic frameworks (COF) case study, Phase I expands the literature to a 1,656-paper corpus and Phase II applies the frozen schema to a 200-paper sample. Compared to ChemDataExtractor, AutoSchema improves coverage and yields more verifiable extractions; compared to an evidence-grounded ChatExtract-style baseline with a manually specified field list, AutoSchema provides stronger evidence support, while fixed-field prompting can reach higher fill rate and domain-specific coverage when the target schema is known. Additional small-scale pilot tests run the freeze-and-reuse pipeline on MOF/ZIF and Alzheimer's disease amyloid/APP processing, and query-generation/retrieval tests on prompt-engineering literature, illustrating AutoSchema's transferability while keeping COF as the main benchmark.
Mingfang Zhu, Yixin Chen, Zhiling Zheng· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
