A Review of Long-Context Processing Ability in Large Language Models: Technical Progress and Challenges

Abstract

In recent years, large language models (LLMs) have achieved significant results in natural language processing. They are applied to various tasks, including text generation, question answering, automatic summarization, code generation, and complex reasoning. With the increasingly complex real scenarios, the length of input text that models need to deal with also grows. Thus, the long-context processing ability of language models has gradually become an important factor in evaluating the practicability of LLMs. This paper gives an introduction to the long-context processing ability of large language models. It first introduces the background of large language models and the basic concept of long-context processing. It then summarizes the main technical methods of long-context modeling, such as improving positional encoding, training stage expansion, inference-stage optimization, and architecture-level innovation. Third, the paper also discusses the use of long-context ability in long-document question answering, long-text summarization, multi-document integration, code understanding and long-context evaluation tasks. Then, summarize the current main challenges and prospects of research work. This paper argues that the ability of long context should not only come from increasing the context window, but also from the ability of the model to locate, integrate and reason about important information in long text.