High-quality large language model inference is increasingly used on mobile edge, from satellites to smart homes, where limited connectivity or privacy requirements motivate local model inference. In many such settings, token generation is inherently compute-limited and added latency may be acceptable, but output qualit...
Jun You, Kun Wang, Jie-Song Chen et al.· Proceedings of the 32nd Annu...· 0 citations
As on-device large language model (LLM) applications like mobile agents become increasingly popular, long-context inference has become a key challenge. The key-value (KV) cache is the foundation of efficient autoregressive decoding, but its memory footprint grows rapidly with context length, often exceeding the memory...
Kun Wang, Jia-Ni Cao, Zhen-Jiang Li· Proceedings of the 32nd Annu...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.