G GDP_pdf is a benchmark built to measure the capabilities of frontier multimodal models inside PDF files, and results for seventeen frontier models on the 100-item benchmark are reported: the best model passes only 30.7% of the items and the worst passes 2%.
Abstract
A large share of day-to-day work in professional domains happens inside PDF files: benefits packets, leases, datasheets, clinical guidelines, construction plans. Benchmarks for document AI have generally measured the required capabilities in isolation: OCR, layout analysis, chart reasoning, table QA, document VQA. A high score on any one of them does not necessarily reveal whether a model can answer a realistic question that someone in the field would actually ask about a specific PDF. GDP_pdf is a benchmark built to measure this directly. It consists of question-document pairs authored by working professionals in ten fields, and a candidate question was kept only when at least two frontier multimodal models failed it in a way that mattered: a wrong answer, missed decisive evidence, or a fabricated claim, rather than a superficial difference such as style. Each item comes with a rubric of atomic criteria, so we can report a graded rubric score as well as a strict task-level pass rate, and each item is tagged against a taxonomy of eleven capabilities in three tiers, spanning text extraction and grounding, table and chart comprehension, cross-referencing, spatial reasoning, and abstention on unsupported queries. We report results for seventeen frontier models on the 100-item benchmark: the best model passes only 30.7% of the items and the worst passes 2%. Most errors trace back to a small set of recurring loss patterns: misaligned tables, misread charts, skipped footnotes and exclusions, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text.
Real-world document tasks often ask professionals to answer questions from annual reports, regulations, clinical guidelines, and technical manuals that span hundreds or thousands of pages. Some questions also require comparing related reports. Reliable long-document understanding is therefore a prerequisite for using LLMs in compliance, clinical, financial, and engineering workflows, where decisions must be traceable to specific evidence pages and the cost of an unsupported answer is high -- yet most existing benchmarks still measure short-context or single-page QA. We introduce XL-DocBench, a fully human-verified benchmark for extra-long document understanding, with 1,519 retained questions from six professional domains and contexts up to 2,303 pages. XL-DocBench goes beyond page-level lookup. 1,103 examples (72.6\%) use multiple evidence pages. The final set also includes 556 questions (36.6\%) that use tables, charts, or figures, and 165 questions (10.9\%) that require evidence from multiple documents. Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format, including 218 None-answer cases. We build the benchmark with a tree-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts. By coupling extra-long professional contexts with page-level evidence and typed rules, XL-DocBench fills a gap left by prior single-page, short multi-page, or text-only long-context benchmarks, and lets future work attribute system failures to retrieval, evidence use, or rule following rather than to a single leaderboard score. The results show that current systems still struggle with long contexts, multi-page evidence, and structured reasoning over professional documents.
Hongchen Wei, Yuanzhe Wang, Bei Liu et al.· 0 citations
A benchmark score credits final answers, but not the route by which an item can be answered. In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding or by benchmark-preserved cues in the wording of answers, non-visual clinical text, visible image text, artificial annotations, or device/context artifacts. We call the resulting score-level overinterpretation reasoning inflation. Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition. Across six medical multimodal MCQ datasets, we separate candidate cues from behavioral evidence through prompt- and image-side audits, modality ablations, and matched repairs that preserve the medical target and answer key. In a 13-configuration open-model panel, full-input accuracy is 62.63%, while text-only and options-only settings achieve 53.96% and 29.71%, respectively. Removing length-gap, absolute/conspicuous, and spatial/prepositional cues lowers accuracy by 6.58, 3.50, and 4.77 percentage points. We also construct MedQA-MM, a 1,000-item shortcut-mitigated subset, where text-only and options-only accuracy fall to 5.21% and 12.33%. This does not imply that models never use images; it shows that medical image-reasoning claims require route-level evidence.
Ben-Lu Wang, Yi-Fan Zhang, Jia-Qing Yu et al.· 0 citations
Professionals across medicine, engineering, finance, manufacturing, and the sciences often make consequential decisions from charts. Existing chart benchmarks do not sufficiently measure this ability: they are dominated by bar, line, and pie formats, rely on shorter reasoning chains, and are nearing saturation, with frontier models already scoring 80-90%. We introduce Chartography, a benchmark of 100 tasks that pair charts drawn from professional practice, in domain-specific formats that standard chart benchmarks rarely include, with questions written by professionals who read these charts for a living and independently verified by three additional experts. In an evaluation of 30 frontier-model configurations (20 scored trials per task), the best configuration reaches only 45.0% mean pass@1; the remainder span 9.0-39.5%. Failures concentrate in visual perception: models can miss nuanced features, misread values along sparsely labeled axes, mishandle projected 3D geometry, and violate domain conventions encoded in the chart. We release all tasks, images, provenance metadata, and evaluation code.
Suhaas Garre, Chris Mutty, Sushant Mehta et al.· 0 citations
Effective question answering over heterogeneous documents requires not only powerful reasoning models but also careful decisions about how evidence is structured and retrieved. In many real-world settings, information is inherently multimodal: a researcher assessing a scientific finding must reason jointly over paragraphs, figures, and result tables; a financial analyst combines textual summaries with tabular data; and a clinician interprets diagnostic imaging alongside patient records. Yet the question of how to decompose such heterogeneous documents into meaningful, retrievable units of evidence remains largely unexplored. In text-only settings, the shift from document-level to passage-level retrieval has demonstrated that retrieval unit granularity is a consequential design choice, with finer-grained units consistently improving both retrieval precision and downstream question answering (QA) performance. However, this line of research has matured almost exclusively in unimodal settings. Current approaches to multimodal retrieval, whether based on joint embedding spaces or modality conversion pipelines, largely inherit the same assumption: that retrieval units are determined by document structure rather than by the informational needs of the query. Multimodal large language models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, but operate over fixed context windows and are costly to deploy at scale, with performance known to degrade when relevant evidence is distributed across long contexts. Across all these paradigms, the design of retrieval units, specifically how heterogeneous documents should be decomposed and grouped into focused, indexable fragments, remains underexplored. This doctoral research addresses the modality representation gap by investigating how multimodal documents should be decomposed for complex question answering. Drawing on the notion of information nuggets, which have long served as atomic units of evidence in QA evaluation, we ask whether operationalising this principle at the indexing stage can improve both retrieval effectiveness and downstream answer quality. We introduce the notion of Composite Chunk Items (CCIs), semantically coherent, self-contained evidence units that may span text, tables, images, and charts, and investigate how their construction, selection, and presentation to an LLM reader affect end-to-end QA performance. A central question is what constitutes the right level of granularity for such units: too fine-grained and coherent evidence is fragmented; too coarse and the noise of document-level retrieval is reintroduced. Preliminary experiments provide early evidence that chunk-level retrieval consistently outperforms document-level baselines across a range of evidence token budgets, motivating a systematic investigation into CCI design as an independent research variable. The proposed research is structured around three directions: how CCIs should be constructed to capture key facts with minimal redundancy; how retrieved chunks should be selected and ranked across heterogeneous modalities; and how evidence should be packaged and presented to maximise LLM reasoning quality. Rather than proposing a new model architecture, this work treats retrieval unit design as a principled empirical research focus, seeking to clarify how structural decomposition of multimodal documents influences robustness, interpretability, and overall QA quality.
Kangheng Liang· Annual International ACM SIG...· 0 citations
Two state-of-the-art multimodal models, Gemma-3 and Qwen-VL, are assessed on their ability to interpret mechanical problem images by eliciting a step-by-step chain of thought (CoT) and a final answer, and final answers are compared to verified solutions to measure accuracy.
Henry Fordjour Ansah, Shreya Banerjee, Pranish Ghimire· 0 citations
The observed performance gains with RAG support the use of LLMs augmented with quality-assured external knowledge in the German healthcare context, while strong performance of open-weight models suggests potential for on-premises deployment in privacy-sensitive clinical environments.
Johannes Schwietering, G. Lichtner· BMJ Health & Care Informatic...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
