We have 4 of 4 papers
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
A cost, quality, and latency Pareto atlas is built to identify the best configurations for different deployment constraints and shows why speed alone is insufficient in vLLM inference, showing why speed alone is insufficient.
This work compares recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch lookahead across chat, agent, and document question answering workloads, and concludes that reuse frequency performs best for agents and document question answering and prefetching does not justify its bandwidth cost.
A router is studied that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class to match the goodput of round robin using six GPUs instead of seven.
Tensor parallelism is the only lever that improves latency (compression makes per-token latency worse), while compression is the only lever that multiplies capacity per dollar (16.5x, against 1.21x for an eightfold spend on GPUs).
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
