Skip to content

Sequential Functional Structured Tucker Compression for Large Language Model Attentions

Sep 2026 · 0 citations · 25 references
Computer Science Mathematics

TL;DR

FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget, achieves the lowest WikiText-2 perplexity among the compared methods.

Abstract

Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.

View source

Similar papers

#artificial intelligence Open access Sep 2026

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

A Context-to-Answer-Aligned Memory Compression (CMC) framework is proposed, which compresses long input contexts into compact Context Memory Embeddings aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights.

Md. Mostafizer Rahman, Md. Faizul Ibne Amin, Md. Shahajada Mia et al. · 1 citation
#machine learning Preprint Oct 2026

Rethinking Sequence Compression with Test-time Regression

TTR-Compressor is evaluated as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models and improves RULER performance over attention-based compressors by a large margin.

Sihan Yang, Jie Huang, Han-Xuan Li et al. · 0 citations
#small language model Open access Oct 2026

Structured Pruning of Small Language Models: An Empirical Study on GQA-Aware Attention and MLP Compression with LoRA Recovery

Large language models have shown strong performance across many natural language tasks, but their size makes deployment difficult on hardware with limited memory and compute. Compression techniques aim to reduce this cost while preserving as much quality as possible. This study investigates structured pruning applied t...

Adnane Bakkou, Badr Hirchoua, M. Banane et al. · 0 citations
#natural language process... Preprint Oct 2026

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Linear attention enables efficient long-context autoregressive decoding by compressing history into recurrent states, but this compression can make selective access to sparse and distant information difficult. Existing chunk-based extensions increase memory capacity, yet learned chunk-mixing coefficients may remain fix...

Zhuo-Kun Chen, Xi Lin, Xi-Yu Wu et al. · 0 citations
#machine learning Preprint Nov 2026

GatedV: Self-Gated Values for Transformer Language Models

Standard attention forms query (Q), key (K), and value (V) matrices through learned linear projections of input hidden states. Architectures such as GAU and Mega incorporate nonlinear transformations into attention. In this paper, we propose GatedV, a simple attention variant that applies SiLU only to V before aggregat...

Li-Ming Yu, Hong-Yu Zhou · 0 citations
#natural language process... Preprint Nov 2026

Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long context...

J. Batista, Emil Vatai, M. Wahib · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.