Differentiable Hierarchical Fusion is presented, a novel framework that merges reasoning models with efficient base models via differentiable optimization to produce concise, accurate outputs and introduces a dual-factor adaptive weighting mechanism to capture intra-block variance and inter-block importance hierarchies, thereby addressing key limitations of static merging heuristics.
Real-world inference tasks for large language models exhibit diverse difficulty levels. Existing LLM serving systems integrate models of different sizes and attempt to route tasks of appropriate difficulty to the most suitable model, aiming to reduce resource waste while guaranteeing service quality. Such systems usually adopt a cascading architecture, which performs inference sequentially from lightweight models to heavyweight models and validates outputs until a model that meets the task requirements is identified. However, when handling complex tasks, the cascading architecture inevitably processes unnecessary small models first, leading to cumulative latency and redundant resource consumption. This paper proposes ParaCascade, a parallel cascading framework that supports early routing. The core idea of ParaCascade is to bypass lightweight models and directly route difficult instances to heavyweight model tiers by pre-estimating task complexity, thus avoiding ineffective computation on lightweight models. In addition, ParaCascade adopts parallel prediction and model parallel inference strategies. At the cost of a slight increase in energy consumption, it significantly reduces the systemic latency caused by sequential processing, thereby improving the overall QoS. Extensive evaluations across diverse workloads on the MMLU-pro and MATH benchmarks show that ParaCascade significantly outperforms both single-model deployments and serial inference serving baselines. While maintaining answer quality, it achieves an inference speedup of 1.16× to 1.51×, demonstrating its superiority in efficient LLM serving systems.
RLCascadeRouter is a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising ``stop''and model selection, and uses trajectory returns and advantages to directly optimize the performance-cost objective.
Results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance, and show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.
CoRun is presented, a scheduling-based system that achieves deterministic inference without requiring batch invariance, and employs isolated prefill and fixed-shape batched decode to handle the two stages of LLM inference, respectively, leveraging CUDA graphs for efficient execution and simplified implementation.
We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
