Skip to content

Context Poisoning as Extreme-Value Attention Interference in Long-Context Language Models

Aug 2026 · 0 citations · 32 references
Computer Science

TL;DR

Under a softmax retrieval abstraction, a finite-sample upper bound is derived showing that maintaining a fixed accuracy target above base rate requires the evidence margin to scale as $\Omega(\sqrt{\log N})$, where N denotes the effective distractor count rather than necessarily the raw context length.

Abstract

Large language models can process increasingly long prompts, yet their ability to locate and use decisive evidence may degrade as irrelevant or confusable context is added. We formulate this phenomenon, which we call context poisoning, as extreme-value interference in attention: the decisive-evidence score is upper-bounded, while the maximum score among effective distractors grows with their number. Under a softmax retrieval abstraction, we derive a finite-sample upper bound showing that maintaining a fixed accuracy target above base rate requires the evidence margin to scale as $\Omega(\sqrt{\log N})$, where N denotes the effective distractor count rather than necessarily the raw context length. The analysis connects long-context degradation to score aliasing, positional aliasing, and softmax dilution. Controlled experiments show that retrieval accuracy decreases as total context grows in the presence of embedded hard negatives, that the same-format condition produces the largest observed accuracy drop among the tested distractor constructions at fixed context length, and that retrieval gating can improve evidence use while its net benefit depends on preserving evidence recall. These results motivate evidence bottlenecks, alias-resistant representations, retrieve-then-reason architectures, verifier-mediated memory, and contrastive anti-poison training.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models

RBS-Attention is proposed, a training-free sparse-prefill method with two complementary selection branches that controls the contribution of rescue blocks while preserving regular block-sparse FlashAttention execution and supports radius-adaptive dual-branch selection as an effective approach to long-context prefill.

Chu-Xu Song, Jiu-Qi Wei, Zhen-Can Peng · 0 citations
#machine learning Preprint Sep 2026

Retrieval Capacity of Self-Attention Under Competition

How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention. Without retraining, we retain only the tokens with the highest attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the s...

Timur Mudarisov, M. Burtsev, Radu State · 0 citations
Preprint Aug 2026

Lexical Perturbations Disrupt LLM Reasoning: An Empirical Study of Attention Diversion

Attention Diversion explains why inference-time strategies, including chain-of-thought prompting, spell-checking, self-repair, and stronger repair models, fail to consistently recover performance: each addresses one channel at a time.

Jiaqi Zhu, Yang Zhang, Jun-Hua Ding et al. · 0 citations
#machine learning Preprint Sep 2026

In-Context Binding Capacity in Language Models

This work derives why interference can lower measured capacity by reducing single-binding recall even when the load-dependent recall profile is unchanged, and characterize capacity at the model's query interface.

Manas Venkata Sai Ravulapalli, Samrath Singh Chadha · 0 citations
#artificial intelligence Conference Open access Oct 2026

Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes

It is found that context does not simply erase memorization, and many samples that are extractable without context remain extractable under the retrieved context, especially as the prefix length increases.

Ali Satvaty, Narjes Sharafi, Jirui Qi et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.