Skip to content
Preprint

Shallow Encoding Emits Deep Memories

Nov 2026 · 0 citations · 96 references
Computer Science

TL;DR

SEED makes the prefill shallow in depth and the decode state shallow in representation, and proves that the open-loop error of the periodic compressions back to low rank is bounded independently of their number.

Abstract

Hybrid language models interleave a few softmax-attention layers with many linear-attention or state-space layers, so every layer keeps a memory of earlier tokens: a key-value cache or a fixed-size recurrent state. Serving them spends more compute and high-bandwidth memory on these memories than the output requires. The prefill runs every layer for every prompt token, and every decode step loads and stores whole recurrent states, which on Qwen3.8-Flash-Next take as many bytes per request as the attention cache of thousands of tokens. We find two properties of trained hybrids that locate this excess. First, writes are easier than reads: writing a deep layer's memory of a token needs little of the deep computation that predicting the next token needs. Trained copies of the deep layers'write paths, which we call emitters, therefore write a prompt's deep memories from the residual stream of a shallower anchor layer, and all prompt tokens but the last run as few as 63% of the layers. Second, recurrent states have a state sink: most of their energy lies in a rank-one term along a fixed direction of each head, and the rest is a low-rank working memory. SEED builds on both: it makes the prefill shallow in depth and the decode state shallow in representation. Each recurrent state is stored as its sink plus low-rank factors, which the Gated DeltaNet, Kimi Delta Attention and Mamba-2 updates extend exactly; we prove that the open-loop error of the periodic compressions back to low rank is bounded independently of their number. SEED keeps the original weights frozen, compresses no attention cache, and runs all layers for every generated token. On six checkpoints of five production hybrids of 30B to over 300B parameters, it is nearly lossless on eleven official exams. Served with SGLang on NVIDIA GB300 GPUs on an agentic-coding trace, it raises the per-GPU throughput of Qwen3.8-Flash-Next 1.2-1.4x.

View source

Similar papers

#machine learning Preprint Oct 2026

Universal Test-Time Training

Recent Test-Time Training (TTT) architectures compress context into fast weights that are updated online and queried as memory. Existing TTT designs keep this memory private to each layer: it recurs only over time, and depth merely indexes L separate memories. We argue that memory ownership need not be tied to depth, a...

Ze-Fan Cai, Qin-Zhe Hu, Ziqiao Ma et al. · 1 citation
Preprint Oct 2026

PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator. Existing state quantizers round the sta...

Patrick Haller, Filipe Laitenberger, Alan Akbik · 0 citations
#artificial intelligence Preprint Oct 2026

Hybrid Latent Attention for Looped Language Models

Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid La...

Yu-Han Chen, Si-Yuan Zhang, Nan Wang et al. · 3 citations · ⚡1
Nov 2026

Pipelining the Memory Hierarchy for Ultra-Long-Context Language Model Training

Long-context language models are attractive because they can read whole documents, code repositories, conversations, and scientific sequences. However, a long inference window is not the same as a model that was trained to use millions of tokens reliably. Direct long-context training is blocked by temporary memory spik...

Jing-Han Yao, Sam Ade Jacobs, Masahiro Tanaka et al. · 0 citations

Low-Discrepancy Dither for Quantized Recurrent State Caches

It is found that a deterministic golden-ratio Weyl dither consistently brings the quantized model closer to the full-precision one than stochastic rounding, across pure and hybrid models, storage formats, and long decoding horizons, at no extra cost.

Snigdha Chandan Khilar · 2 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.