Skip to content

Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction

Jul 2025 · arXiv.org · Vol abs/2507.16271 · 0 citations · 49 references
Computer Science

TL;DR

AOE (Arranged and Organized Extraction), a bilingual benchmark that evaluates LLMs'ability to distill long-form, multi-document inputs into structured tables, is introduced, revealing that structured extraction from long contexts remains a fundamental challenge - an"Area of Effect"(AOE) that impacts all tested systems.

Abstract

Large language models (LLMs) are increasingly expected to extract and synthesize structured information from complex real-world documents, yet they typically produce unstructured, hard-to-verify paragraph-style outputs. We introduce AOE (Arranged and Organized Extraction), a bilingual benchmark that evaluates LLMs'ability to distill long-form, multi-document inputs (64k to 1.8M tokens) into structured tables. AOE comprises 12 tasks across four domains (academic, financial, legal, and medical), each grounded in genuine analytical needs, with schema guidance provided to isolate extraction quality from formatting variance. We evaluate 12 frontier LLMs (both open-weight and proprietary) under four generation settings of increasing autonomy: direct full-document input, RAG-based retrieval, offline document-browsing agent, and online web-search agent. Even the best-performing model achieves only 42.7% Cell F1, and retrieval-augmented approaches often degrade rather than improve performance, revealing that structured extraction from long contexts remains a fundamental challenge - an"Area of Effect"(AOE) that impacts all tested systems.

View source

Similar papers

Open access Oct 2026

Benchmarking Local Large Language Models for Structured Knowledge Extraction in Conversational Commerce

The integration of natural language understanding into conversational commerce requires robustly extracting structured intents from idiomatic text. While cloud-based models are highly capable, their latency, privacy risks, and operational costs necessitate efficient local alternatives. To address this, we empirically e...

Ratomir Karlović, Luka Sever, Marijela Miličević et al. · 0 citations

MERGE: Multi-LLM Ensemble for Retrieval via Generative Enrichment

Large Language Models (LLMs) are increasingly used to enrich user queries in information retrieval (IR) so that a standard retriever such as BM25 can bridge vocabulary gaps with the target corpus. Any single LLM, however, is limited by its training data and architectural biases, and its enrichment behavior depends on h...

Tzu-I Ho, Yung-Yu Shih, Shang-Yu Su et al. · 0 citations
Preprint Oct 2026

FormBench: A Benchmark for Fine-Grained Retrieval in Formulation Patents

We introduce FormBench, a platform for testing the ability of models to retrieve precise chemical information from technical documents. FormBench contains 55,352 queries and two retrieval corpus variants each with ~1M passages: FormBench-Random (C0), with randomly sampled distractors, and FormBench-Structured (C1), wit...

Vineeth Venugopal, Elsa A. Olivetti, J. Welch · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.