Industrial-Instruction addresses the gap in public instruction-tuning or benchmark datasets built from real industrial technical reports, and releases two parallel versions built by the same pipeline, enabling a direct comparison of open- versus frontier-model data generation.
Abstract
Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.
LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost, and is the first to score value accuracy, record completeness at scale, grounding, and measured cost together.
Boyang Zhang, Adrian Lyjak, Elizabeth Stewart et al.· arXiv.org· 1 citation
This study develops a multi-source Retrieval-Augmented Generation (RAG) based Question Answering (QA) system that automatically integrates heterogeneous knowledge sources through a unified source parameter to enhance knowledge transfer and question answering for organizational support and employee onboarding.
Krisna Dwi Setya Adi, Ivan Michael Siregar· Jurnal Ragam Pengabdian· 0 citations
On-site management for technical supervision for power systems is implemented based on diversified unstructured documents including industrial technical specifications, corporate operational codes and fault incident summaries. Conventional keyword-based retrieval suffers from prominent drawbacks such as semantic mismatch, inconsistent alignment between regulatory clauses and search queries, as well as poor precision in fault case lookup. Targeting endto- end practical deployment of domain-specific retrieval tailored to technical supervision, this paper develops an integrated technical pipeline covering structured parsing of heterogeneous multi-source technical documents, automatic compilation of domain question-and-answer corpora, dual-layer human-machine quality control, and fine-tuning of embedding and reranking dual models. To begin with, hierarchical structured extraction rules are formulated for 6 categories of unstructured materials: industry technical standards, internal corporate circulars, implementation guidelines and detailed rules, alongside typical fault investigation reports. Large language models are leveraged to automatically generate paired datasets linking regulatory provisions with corresponding inquiry items, followed by noise reduction via a two-tier quality control mechanism combining preliminary automated screening and selective manual inspection. Next, domain-specific parameter fine-tuning is performed on Qwen3-Embedding-0.6B dense retriever and Qwen3-Reranker- 0.6B reranking model to build a two-stage retrieval framework consisting of coarse-grained dense candidate retrieval and subsequent refined result reranking. Comparative experiments are conducted on an in-house Q&A benchmark for technical supervision and publicly available universal power retrieval datasets. Quantitative results across Recall@k metrics verify that the proposed approach outperforms prevalent baseline alternatives comprehensively and achieves state-of-the-art performance within this niche application field.
Sai Zhang, Xiao Liang, Bochuan Song et al.· International Conference on...· 0 citations
Consistency verification of drilling reports is critical for engineering quality control because a single data item may be distributed across reports with different formats, units, and page structures. Existing retrieval-augmented generation methods remain sensitive to retrieval and parsing errors in such documents, whereas ultra-large models impose substantial local computing and memory costs. This study proposes a lightweight tool-augmented framework based on dual-teacher distillation and sparse mixture-of-experts (MoE) modeling. Qwen3-235B-A22B serves as the primary teacher and Qwen3-30B-A3B as the assistant teacher. Their tool-use and task-planning capabilities are transferred to a sparse MoE student upgraded from a Qwen3-1.7B dense backbone through trajectory pruning, sample decomposition, and token-level Kullback–Leibler (KL) distillation. The student adopts an eight-expert Top-2 routing architecture. Experiments on 1000 drilling reports containing 30,127 verification instances show an F1 score of 58.0 ± 0.5%, with file-level, location-level, and exact-match accuracies of 66.5%, 55.2%, and 45.0%, respectively. The model contains 9.1B total parameters and 2.8B activated parameters, and reaches a latency of 12.1 ms per forward pass and a memory footprint of 18.4 GB under bfloat16 (BF16) precision. The reported F1 score characterizes the end-to-end verification task rather than an autonomous safety decision capability. The framework is intended to support evidence localization, anomaly prioritization, and expert review in local deployment settings.
Scientific papers require models to reason jointly over text, equations, figures, tables, code, and datasets while preserving the provenance of supporting evidence. Existing benchmarks typically evaluate these capabilities in isolation, leaving unclear whether multimodal models can support realistic scientific-reading workflows. We introduce SciDocBench, a workflow-centered benchmark for scientific document understanding. It contains 124 expert-authored and difficulty-screened questions organized into seven research-assistant capability groups and 19 subtasks across five scientific domains. Each question is instantiated under four matched conditions combining English or Chinese questions with all-images-first or interleaved document representations, yielding 496 evaluation instances for controlled analysis. The strongest evaluated system achieves only 62.6/100, with pronounced weaknesses in document perception, evidence grounding, verification, and cross-document reasoning. To translate these diagnostics into scalable training signals, we introduce SciDocIR, a typed evidence-graph representation that preserves scientific document objects, layout and cross-reference relations, and provenance. Building on SciDocIR, we construct SciDocDataset, comprising approximately 15K supervised fine-tuning samples and 8K reinforcement-learning samples across 14 verifiable subtasks. Together, SciDocBench, SciDocIR, and SciDocDataset form an evaluation-to-training framework for diagnosing and improving scientific-document assistants. The project page is available at https://github.com/InternLM/SciDocBench.
Shenxi Wu, Yu-Hong Liu, Haosong Zhang et al.· 0 citations
Real-world document tasks often ask professionals to answer questions from annual reports, regulations, clinical guidelines, and technical manuals that span hundreds or thousands of pages. Some questions also require comparing related reports. Reliable long-document understanding is therefore a prerequisite for using LLMs in compliance, clinical, financial, and engineering workflows, where decisions must be traceable to specific evidence pages and the cost of an unsupported answer is high -- yet most existing benchmarks still measure short-context or single-page QA. We introduce XL-DocBench, a fully human-verified benchmark for extra-long document understanding, with 1,519 retained questions from six professional domains and contexts up to 2,303 pages. XL-DocBench goes beyond page-level lookup. 1,103 examples (72.6\%) use multiple evidence pages. The final set also includes 556 questions (36.6\%) that use tables, charts, or figures, and 165 questions (10.9\%) that require evidence from multiple documents. Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format, including 218 None-answer cases. We build the benchmark with a tree-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts. By coupling extra-long professional contexts with page-level evidence and typed rules, XL-DocBench fills a gap left by prior single-page, short multi-page, or text-only long-context benchmarks, and lets future work attribute system failures to retrieval, evidence use, or rule following rather than to a single leaderboard score. The results show that current systems still struggle with long contexts, multi-page evidence, and structured reasoning over professional documents.
Hongchen Wei, Yuanzhe Wang, Bei Liu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
