AOE (Arranged and Organized Extraction), a bilingual benchmark that evaluates LLMs'ability to distill long-form, multi-document inputs into structured tables, is introduced, revealing that structured extraction from long contexts remains a fundamental challenge - an"Area of Effect"(AOE) that impacts all tested systems.
Abstract
Large language models (LLMs) are increasingly expected to extract and synthesize structured information from complex real-world documents, yet they typically produce unstructured, hard-to-verify paragraph-style outputs. We introduce AOE (Arranged and Organized Extraction), a bilingual benchmark that evaluates LLMs'ability to distill long-form, multi-document inputs (64k to 1.8M tokens) into structured tables. AOE comprises 12 tasks across four domains (academic, financial, legal, and medical), each grounded in genuine analytical needs, with schema guidance provided to isolate extraction quality from formatting variance. We evaluate 12 frontier LLMs (both open-weight and proprietary) under four generation settings of increasing autonomy: direct full-document input, RAG-based retrieval, offline document-browsing agent, and online web-search agent. Even the best-performing model achieves only 42.7% Cell F1, and retrieval-augmented approaches often degrade rather than improve performance, revealing that structured extraction from long contexts remains a fundamental challenge - an"Area of Effect"(AOE) that impacts all tested systems.
The integration of natural language understanding into conversational commerce requires robustly extracting structured intents from idiomatic text. While cloud-based models are highly capable, their latency, privacy risks, and operational costs necessitate efficient local alternatives. To address this, we empirically e...
Ratomir Karlović, Luka Sever, Marijela Miličević et al.· Information· 0 citations
Overall, the proposed KV-ER Hybrid representation improved retrieval and answer generation under the evaluated OCR–RAG configuration without modifying the OCR model, retriever, or generator.
Large Language Models (LLMs) are increasingly used to enrich user queries in information retrieval (IR) so that a standard retriever such as BM25 can bridge vocabulary gaps with the target corpus. Any single LLM, however, is limited by its training data and architectural biases, and its enrichment behavior depends on h...
Tzu-I Ho, Yung-Yu Shih, Shang-Yu Su et al.· arXiv.org· 0 citations
We introduce FormBench, a platform for testing the ability of models to retrieve precise chemical information from technical documents. FormBench contains 55,352 queries and two retrieval corpus variants each with ~1M passages: FormBench-Random (C0), with randomly sampled distractors, and FormBench-Structured (C1), wit...
Vineeth Venugopal, Elsa A. Olivetti, J. Welch· 0 citations