Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference
InplaceKVCache is proposed, the first KVCache abstraction whose format fixes each byte's physical residency at write time, so that the CPU--GPU load balance can be adjusted without moving data after placement, turning load balancing into pure scheduling.
En-Da Yu, De-Zun Dong, Xiang-Ke Liao
· 0 citations