Data Foundations of Long-Context Language Models: A Survey

A Survey on Context Injection Strategies for Long-Context Language Models: Three Perspectives

The performance of large language models (LLMs) is fundamentally shaped by what information reaches them at inference time. As context windows expanded, the challenge shifted from can we fit the data to how do we fit the right data, efficiently and faithfully. This survey argues that context injection strategy, rather than context capacity, is the defining research challenge for long-context LLM deployment. We organize the literature into five major families - direct injection, compression-based injection, retrieval-based injection, memory-based injection, and hybrid/agentic injection. Our primary contribution is a three-axis analytical framework revealing that injection performance is jointly governed by selection (what information is chosen), representation (how it is encoded), and scheduling (when it is delivered during inference). Systematic analysis of the surveyed systems shows that current approaches overwhelmingly optimize the selection axis while representation and scheduling remain largely unexplored. We further identify six structural open challenges that are unlikely to be resolved within any single paradigm, and discuss evaluation gaps that current benchmarks fail to address. These findings suggest that progress on representation and scheduling axes, rather than further scaling of context windows, is the most promising direction for closing the gap between nominal and effective context capacity in deployed LLM systems.

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

PredicateLongBench is proposed, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints drawn from a broader predicate class.

Towards Conversational Dataset Retrieval: A Survey

Large language models (LLMs) have sparked renewed interest in Conversational Information Retrieval (CIR). Within this shift, Conversational Dataset Retrieval (CDR) is emerging as a new subfield that focuses on using natural, context-aware dialogue to discover structured and semi-structured datasets. We present the first integrative review of this rapidly evolving landscape. We synthesise insights from 44 publications spanning user studies, conversational system design choices, dataset representation and access, and evaluation methods. We introduce a novel, layered conceptual framework that organizes research in CDR across four key dimensions: User Layer, System Layer, Data Layer, and Evaluation Layer. This framework is used to highlight recurring design patterns, technical advances, and persistent gaps. Our analysis identifies core challenges in CDR, including the lack of standardized evaluation benchmarks and limited support for ambiguous or evolving user intent. The aim of this survey is to provide a structured foundation for future research on CDR and guide the development of more interactive, intelligent, and user-centric dataset retrieval systems.

A comparative review of modern large language model paradigms: GPT-4, BERT, Gemini, and DeepSeek

Comparison of GPT-4, BERT (bidirectional encoder representations from transformers), Gemini, and DeepSeek large language models (LLM), focusing on architectures, training methodologies, and real-world applications reveals GPT-4 excels in natural language generation and complex reasoning, supporting up to 128K tokens with moderate latency and higher costs making it effective for conversational artificial intelligence (AI).

MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like ``however''or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.

Pruning large language models: a systematic literature review