LLM-based agents execute long-horizon tasks through repeated model calls interleaved with tool execution and user interaction. As each call extends the history accumulated in previous turns, prefix caching avoids repeated prefill of the agent's entire context. However, the aggregate cache footprint grows with context l...
Zai-Feng Pan, Chris Wu, Zheng-Ding Hu et al.· 0 citations
LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimizations is challenging because identical tasks can produce different execution trajectories. Changes in generated tokens can alter subsequent...
Zai-Feng Pan, Michael Wang, Chris Wu et al.· 0 citations
This work proposes a series of optimizations for these two kernels, including computation-transfer pipelining, load balancing, and memory access fusion, achieving 1.97 × to 2.16 × proof generation speedup over a state-of-the-art open source GPU acceleration library.
Xinwei Qiang, Liukun Yu, Xiyu Wang et al.· IEEE International Symposium...· 0 citations
Block drafters propose several tokens in one forward pass, before earlier target tokens are realised. Their rejection mixes two losses: missing within-block path information and imperfect modelling of observable information. Accepted length cannot distinguish them. We separate the two with an information floor, the min...
Xinwei Qiang, Xiang Fang, Changli Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
