Experiments on synthetic and real-world benchmarks show that FG$^2$-GDN and its variant improve associative recall and long-context understanding over GDN and KDA, with comparable computational efficiency.
Ping-Wei Sun, Yuxuan Hu, Jian-Chao Tan et al.· arXiv.org· 1 citation· ⚡1
Decay-Aware State Compression (DASC), which derives retention horizons from model weights, selects long-horizon state units, and packs them into a ragged state checkpoint layout to integrate efficiently with tensor-parallel inference engines.
Yanzhi Yu, Ping-Wei Sun, Jian-Chao Tan et al.· 0 citations
DAMP uses both quantization-error energy and decay-based persistence to identify high-risk channels during offline calibration and stores these channels at higher precision and the remainder in INT8, the first to study post-training quantization of recurrent states in GDN and KDA based language models.
Tao Zhang, Jian-Chao Tan, Ping-Wei Sun et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.