KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises of this hierarchy. First, bimodal reuse defeats DRAM temporal filtering, since workloads are dominated either by one-sho...
Xin Huang, Teng Ma, Zhuo-Ran Li et al.· 0 citations
Tessera is presented, the first kernel disaggregation system to improve performance and cost efficiency on heterogeneous GPUs for large model inference, and shows that a heterogeneous GPU pair under Tessera can even exceed the throughput of two homogeneous high-end GPUs at a lower cost.
Tian-Cheng Hu, Jin Qin, Zheng Wang et al.· arXiv.org· 0 citations
Large language models now write a growing share of the world's code, increasingly inside agents and serving systems that compile, execute, or dispatch generated code without line-by-line review. This works well for mainstream languages but remains brittle for low-resource programming surfaces such as domain-specific la...
Shuoming Zhang, Ruiyuan Xu, Haofeng Li et al.· arXiv.org· 0 citations
StrataCL introduces registration-on-allocation to realize user-buffer direct communication, and designs communication operators with workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to exploit supernode architecture features.
Tian-Cheng Hu, Jin Qin, Yu-Zheng Wang et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
