DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

Dengke Han

Jul 2026 · arXiv.org · Vol abs/2607.24434 · 0 citations · 26 references

Computer Science

TL;DR

DraftExpert is proposed, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference that improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.

Abstract

Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence--expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

EdgeXpert is proposed, a software-hardware co-designed LLM accelerator that resolves this incompatibility and achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.

Sangwoo Ha, Hyunwoo Seo, Yurim Jo et al. · 0 citations

#artificial intelligence Preprint Aug 2026

S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama$.$cpp, S2-MoE achieves up to $5.3\times$ speedup (about $2.0\times$ on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.

Haochen Huang, Shengxuan Qiu, Meng Li · 0 citations

Preprint Jul 2026

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

Results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models, and a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection is proposed.

Jincheng Xie, Runheng Liu, Heyan Huang et al. · 1 citation

Jul 2026

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

A unified efficiency analysis is presented showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost, and SparseSpec-L, a training-free self-speculative decoding framework for long-context inference is introduced.

Yue Liu, Yuan Zeng, Min Lyu et al. · arXiv.org · 0 citations

Preprint Aug 2026

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

ExactMoE, an inference design that applies symmetric group-128 four-bit weight quantization only to routed experts, stores those experts in kernel-native MARLIN form in pinned host memory, and executes all selected experts through a configurable GPU-resident slot cache and fused grouped MoE kernels, identifies a practical memory-transfer-throughput frontier for complete-expert MoE inference.

Amjad Saab · 0 citations

Characterizing self-speculative decoding approaches for accelerating LLMs

Five self-speculative decoding techniques are characterized across three model sizes and three datasets and recommendations for future research in this area are provided.

Jungmin Ha, Karthik Ganesan, Anh Nguyen et al. · 0 citations