TL;DR
It is argued that LLM-native data systems should expose selected inference-time mechanisms to the optimizer as physical design choices and advocate constructing Pareto frontiers of candidate implementations and exposing only non-dominated choices to the optimizer.
A framework for scalable, dataset-scale LLM-based data transformation that processes data row-by-row in batches over lazily evaluated, distributed dataframes, enabling LLM-based transformations on arbitrarily large datasets without loading them into memory.
This work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.
Kalypso is presented, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm, demonstrating that query-aware LLM serving can substantially improve the efficiency of semantic query execution.
The results suggest that the sweet spot for small models in large-model inference systems lies not in solving complex tasks independently, but in performing lightweight, structured, and verifiable auxiliary operations.
This work presents InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state, and encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV.
This model reveals one key opportunity: dividing a restore request proportionally between the storage path and the GPU can improve inference performance while still meeting SLOs, and reduces the KV-cache storage stack to a performance model based on per-tier capacity, per-tier and interconnect bandwidth, and GPU arithm...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
