ACache is presented, a cross-request cache reuse mechanism for shared spans, or affixes, at any position: prefix, infix, or suffix, and co-designs it with a modern inference engine, whose attention reads each request's recomputed Anchor KV states alongside one affix cache shared across concurrent requests.
Kai-Huan Liang, An Zhong, Xin Tan et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.