JetSpec is proposed, a head-based SD framework that combines one-forward drafting efficiency with branch-wise causal conditioning that enables JetSpec to convert larger draft budgets into longer accepted prefixes and higher end-to-end speedup.
KV-Invariant Transformer Expansion (KITE) is introduced, a scaling paradigm that trains the model from a smaller size to a larger size (i.e., saving training costs via upcycling), while places newly added parameters in regions that do not affect attention KV.
Zhi-Heng Hu, Yi-Xun Wei, Jian Zhou et al.· 0 citations
This work introduces Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales, and provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training.
Houyi Li, Wen-Zheng Zheng, Qiufeng Wang et al.· Neural Information Processin...· 4 citations· ⚡1
MISA-T, a routing-layer admission policy for mixed rollout serving that combines adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting, is presented.
Zetao Hong, Song Yuan, Yuan-Hao Ding et al.· 2 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.