The rational design of high-performance catalysts is pivotal for modern industry but is often hindered by the time-consuming nature of trial-and-error methods. While computational and data-driven approaches have accelerated materials discovery, significant challenges remain, especially for complex systems like multie...
Hybrid large language models combine full-attention and recurrent linear layers to reduce inference costs. Prefix caching can further reduce redundant computation across requests, but a recurrent checkpoint cannot recover an earlier state corresponding to a shorter prefix. Existing hybrid prefix caching systems retain...
Zai-Feng Pan, Lu-Jun Wang, Xin-Wei Qiang et al.· 0 citations
Megakernels let one persistent GPU program prepare dependent operators early and execute ready tasks together. We study the benefits and costs of these capabilities by reconstructing two LLM megakernels, ThunderKittens (TK) and VDCores, as separate kernels on H100 using their original operator code. Our results show th...
Xin-Wei Qiang, Zi-Yi Xu, Chang Chen et al.· 0 citations
Large Language Models (LLMs) offer an adaptable alternative to traditional quantum compilation heuristics, which bottleneck progress by demanding costly manual redesigns whenever hardware evolves. However, naive single-pass LLM approaches fail under the unique challenges introduced by quantum compilation. We introduce...
Ji-Xuan Ruan, Zhuo Cui, Zheng-Ding Hu et al.· 0 citations
Sparse attention limits the history read by each query, but serving long contexts still requires substantial GPU memory. Existing KV offloading systems retain the full history of indexer keys on GPU, limiting decode concurrency. Prefill's temporary memory demand can further delay admission even when enough memory remai...
Zai-Feng Pan, Chris Wu, Xin-Wei Qiang et al.· 0 citations
Zepp is presented, which directly optimizes the bottleneck communication in distributed MoE serving subject to simplified balance constraints on physical resources, i.e., GPUs and NICs and progressively optimizes inter-node communication across placement, routing, and execution.
Chang Chen, Andrew Yang, Tian-Cheng Chen et al.· 0 citations
This work presents RR-EVICT, a fine-grained prefix-cache eviction strategy that distributes reclamation across idle agent trajectories, and implements RR-EVICT in SGLang and evaluates conversational and coding-agent workloads under colocated and prefill-decode-disaggregated serving.
Zai-Feng Pan, Chris Wu, Zheng-Ding Hu et al.· arXiv.org· 2 citations
The experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.
Zai-Feng Pan, Michael Wang, Chris Wu et al.· arXiv.org· 4 citations
Block drafters propose several tokens in one forward pass, before earlier target tokens are realised. Their rejection mixes two losses: missing within-block path information and imperfect modelling of observable information. Accepted length cannot distinguish them. We separate the two with an information floor, the min...
Xinwei Qiang, Xiang Fang, Changli Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.