PredicateLongBench is proposed, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints drawn from a broader predicate class.
Abstract
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness. Notably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PredicateLongBench, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines - a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PredicateLongBench, though challenging, are conceptually simple and do not require LLM-based generations or judges.
Despite the rapid progress of LLMs, their evaluation remains hindered by static, manually curated benchmarks with limited task coverage and poor adaptability to emerging domains. Existing automated approaches typically operate within fixed task schemas and often fail to autonomously discover new evaluation dimensions, limiting both scalability and effectiveness. To address these gaps, we propose A UTO T ASK E VAL , an automated framework that constructs domain-specific benchmarks directly from unstructured corpora. Using a refined Bloom’s Taxonomy, the framework systematically discovers tasks, enriches contextual grounding via iterative Socratic prompting, and generates diverse, progressively challenging evaluation instances. Applied to the complex and knowledge-intensive legal domain, A UTO - T ASK E VAL uncovers a broader and more fine-grained task space than expert-curated benchmarks while producing high-quality instances that preserve established model-level evaluation trends. We further validate its robustness in a low-structure e-commerce review domain. Together, these results show that A U - TO T ASK E VAL enables scalable, adaptive, and high-fidelity LLM assessment across domains and model families, advancing autonomous and capability-sensitive evaluation.
Qingqing Lyu, Linjuan Wu, Yongliang Shen et al.· Annual Meeting of the Associ...· 0 citations
As the context window of Large Language Models (LLMs) continues to expand, the data required to effectively train and evaluate these capabilities remains underexplored. With existing research primarily focuses on architectural optimization, there is a need for a systematic, data-centric review. This survey bridges this gap by investigating the data foundations of Long-Context Language Models (LCMs). We begin by examining current data strategies alongside their strengths and limitations, mapping the required data to desired model capabilities. Building on this, we explore how targeted training data designs drive core, often interconnected skills such as retrieval, reasoning, and aggregation. Furthermore, we analyze the evaluation landscape, illustrating how selecting appropriate benchmarks is crucial for probing capability boundaries and guiding effective model selection. Finally, we synthesize actionable guidelines for data construction and outline critical future directions to propel the advancement of long-context language models, including quantifying data quality, establishing scaling laws for length distributions, and developing dynamic evaluation frameworks.
Zechen Sun, Yuyang Sun, Zhao-yu Su et al.· Transactions of the Associat...· 0 citations
Large Language Models (LLMs) have achieved remarkable success in temporal reasoning. However, existing benchmarks predominantly adopt a "single-answer" paradigm, focusing on verifying the existence of a specific fact while overlooking the challenge of exhaustiveness. In real-world scenarios, entities often simultaneously play multiple roles or exist in multiple states within the same timeframe. To bridge this gap, we introduce MulTR, a comprehensive benchmark designed for multi-answer temporal reasoning from long unstructured contexts. Specifically, MulTR integrates structured temporal facts from Wikidata and natural language text from Wikipedia MulTR integrates structured temporal facts from Wikidata and natural language text from Wikipedia through a logic-driven synthesis process. Notably, we formulate two distinct settings, question-dependent and document-dependent, based on the presence of cue words in the question. It is designed to systematically decouple temporal reasoning capabilities from the uncertainty of the number of answers. Experiment results demonstrate that state-of-the-art models suffer from retrieval laziness, terminating the search process prematurely after locating the first valid piece of evidence. Consequently, their performance drops sharply when evaluated on strict exact match metrics. MulTR, as a diagnostic testing platform, reveal these defects and establish the rigorous standard for future research in dynamic knowledge processing. The MulTR benchmark and evaluation prompt are publicly available at https://github.com/TemporalNLP/MulTR.
Shaojuan Wu· Annual International ACM SIG...· 0 citations
The performance of large language models (LLMs) is fundamentally shaped by what information reaches them at inference time. As context windows expanded, the challenge shifted from can we fit the data to how do we fit the right data, efficiently and faithfully. This survey argues that context injection strategy, rather than context capacity, is the defining research challenge for long-context LLM deployment. We organize the literature into five major families - direct injection, compression-based injection, retrieval-based injection, memory-based injection, and hybrid/agentic injection. Our primary contribution is a three-axis analytical framework revealing that injection performance is jointly governed by selection (what information is chosen), representation (how it is encoded), and scheduling (when it is delivered during inference). Systematic analysis of the surveyed systems shows that current approaches overwhelmingly optimize the selection axis while representation and scheduling remain largely unexplored. We further identify six structural open challenges that are unlikely to be resolved within any single paradigm, and discuss evaluation gaps that current benchmarks fail to address. These findings suggest that progress on representation and scheduling axes, rather than further scaling of context windows, is the most promising direction for closing the gap between nominal and effective context capacity in deployed LLM systems.
Aicha Dakir, Mohamed El Hajji, Tarek Ait Baha et al.· EPJ Web of Conferences· 0 citations
Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.
Abhigya Verma, Khyati Mahajan, Amit Kumar Saha et al.· 0 citations
Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.
Xinyu Zhu, Zhenqin Xu, Xiaohan Wei et al.· 0 citations