When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
A two-term cost model means that hits on prompts shorter than about 17k tokens cannot beat recompute at any bandwidth, and shard the MLA KV store across PCP ranks raises prefill throughput by 14%, and by 34% together with an upstream kernel fix.
M. Ibrahim
· 0 citations