HyperCut: Fast Inter-Layer Scheduling via Directed Hypergraph and Early Filtering

RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention

Dynamic sparse attention reduces long-context prefill cost by routing each query chunk to a small set of key chunks at every Transformer layer. The sparse attention kernel avoids most token interactions, but the router still rebuilds a chunk--chunk score matrix layer after layer, even when the selected routes change little. We introduce RouteRelay, a router-agnostic method that reuses only route metadata across depth while continuing to compute attention with the current layer's queries, keys, and values. Anchor layers perform full routing. Intermediate layers rescore the previous top-$k$ route and a compact sentinel set of near-miss and randomly probed chunks. A query row is rerouted only when a sentinel challenges its weakest selected chunk. We give a top-$k$ stability condition, a probabilistic bound on missed challengers, and a row-selective GPU execution design. In a reproducible empirical evaluation, RouteRelay retains at least 99.99% route recall while rerouting 25.0%, 55.4%, and 78.2% of rows under low, moderate, and high cross-layer drift, respectively. Across routing scales, RouteRelay retains 100.0% recall while evaluating 38.4--51.6% of full-routing score pairs as the key-chunk count grows from 128 to 1024. Its unfused CPU execution remains slower than dense matrix multiplication, exposing row compaction and ledger updates as the main kernel-engineering targets.

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

This work presents a fine-grained approach that overlaps expert compute with the second all-to-all via tile-level signaling and scheduling, making it practical for improving distributed MoE execution efficiency on multi-GPU systems.

DAN-Scheduler: Deterministic Three-Stage Co-Optimization of Scheduling, Memory Layout, and Pipeline Overlap for General-Purpose NPUs

CrossServe: Cross-Layer Scheduling for SLO Optimization in Multi-Tenant LLM Serving

The deployment of Large Language Models (LLMs) as multi-tenant cloud services is now widespread, but maintaining high Service Level Objective (SLO) attainment across diverse tenants remains challenging. Current serving systems focus on a single layer of the stack, either using iteration-level batching or coarse-grained instance-level routing. This approach causes significant SLO violations during bursty workloads and struggles to balance throughput with differentiated tenant priorities. The core issue lies in intra-instance schedulers lacking awareness of the global cluster state, while cluster-level routers do not account for fine-grained, per-iteration execution behavior within each GPU. We introduce CrossServe, a cross-layer scheduling framework that jointly optimizes request routing, adaptive micro-batching, and tenant-weighted preemption to maximize SLO attainment in multi-tenant LLM serving. CrossServe integrates three complementary layers: (1) an online, lightweight length classifier that separates requests to reduce Head-of-Line (HoL) blocking; (2) an SLO-aware adaptive micro-batching system that dynamically tunes prefill chunk sizes based on the real-time Time-Per-Output-Token (TPOT) slack of decoding requests; and (3) a global routing and preemption layer that distributes load across GPU instances while using tenant weights when reclaiming resources under overload. We evaluate CrossServe with BurstGPT-derived trace replay on a 32-GPU cluster. Compared with vLLM, Sarathi-Serve, and SOLA, CrossServe increases overall SLO attainment by 53, 29, and 14 percentage points respectively (from 42% to 95%, from 66% to 95%, and from 81% to 95%) while sustaining high throughput. In the ablation workload, removing global routing raises P99 TTFT from 185 ms to 410 ms; the full system therefore reduces this tail latency by 54.9% relative to that ablation and improves the observed throughput–SLO tradeoff.

Turbo: Efficiently Serving Long-Context Large Language Models with In-Network Aggregation

This work proposes Turbo, a first-of-its-kind in-network aggregation system that accelerates long-context inference by offloading query broadcast and attention aggregation to switches and introduces a rolling forward scheme that propagates states to enable cross-stage updates.

A Bipartite-graph Guided DNN Weight Remapping Scheme for Achieving Low-cost MRAM CIM Reprogramming

Compute-in-memory (CIM) architectures mitigate the von-Neumann bottleneck by embedding computation directly within memory crossbars, delivering order-of-magnitude improvements in energy efficiency and throughput. Among the emerging technologies, non-volatile-memory (NVM)-based CIM is particularly attractive owing to its non-volatility, ultralow standby power, and high integration density. Nevertheless, capacity-constrained NVM crossbars can accommodate only a fraction of a deep neural network's weights at any instant, necessitating frequent reprogramming that incurs substantial latency and energy penalties. We observe that the problem of scheduling weight updates onto such a limited-capacity crossbar is isomorphic to a bipartite-graph assignment problem. Motivated by this insight, we propose a static scheduling framework that systematically minimizes reprogramming cost. The framework comprises three synergistic stages: (i) bipartitegraph-driven row- and column-reordering formulated as an optimal assignment problem; (ii) fine-grained bit-flip minimization guided by hardware-aware correction logic; and (iii) a prioritydriven consolidation layer that allows designers to favor either energy or latency. Evaluations conducted on a customized STT-MRAM CIM simulator demonstrate an average 41% reduction in programming energy, an average 19% reduction in latency, and a 49% improvement in the energy-delay product over the conventional baseline.