This work introduces LongSpark, a block-diffusion drafter that achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state.
Abstract
Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.
This analysis shows that verifier skipping is a useful new lossy axis and, surprisingly, its key challenge is prefix scheduling rather than token prediction alone.
DEdit is introduced, a diffusion-based drafter that can not only draft by conventional parallel unmasking but also iteratively edit its draft through token-to-token predictions, and ProposalMix, a training scheme that mixes draft predictions with ground-truth tokens based on first-pass confidence during training.
Long-Xuan Yu, Bingsen Chen, Peng Shi et al.· arXiv.org· 0 citations
Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-specu...
Chen-Ce Yang, Ning-Xi Cheng, Arash Akbari et al.· 0 citations
LiLiCorr, a Lightweight Likelihood-based model that Correlates the per-position marginal distributions a drafter already produces, is introduced, a Lightweight Likelihood-based model that Correlates the per-position marginal distributions a drafter already produces.
M. Rusanovsky, Yoav Miron, Roy Uziel et al.· 4 citations
NebastianSD is presented, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools.
Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as context grows and draft acceptance declines. We focus on attention-mass dilution: as softmax normalizes over more visible Keys, the mass concentra...
Jeonghoon Park, Seo-An Jo, Jongwon Lee et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.