Linear attention and recurrent models compress past tokens into a fixed-size state that is updated at every step. However, storing these states in high precision incurs substantial memory and bandwidth costs at large batch sizes, motivating low-bit state quantization. Unlike weights and key--value caches, a recurrent s...
Hao-Tian Li, Li-Ran Dong, Kai-Cheng Xiao et al.· 0 citations
KV-cache quantization and linear attention are two representative approaches to tackling the storage and computational costs of Transformers. KV-cache quantization compresses individual KV entries into discrete codes but retains all entries, whereas linear attention recurrently aggregates multiple historical KV contrib...
Kai-Cheng Xiao, Li-Ran Dong, Hao-Tian Li et al.· 0 citations
SkillVine is proposed, an automatic skill-evolution framework that formulates skill evolution as a graph search problem and employs a branching exploration strategy, and achieves a balance between exploration and exploitation.