Skip to content

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

Sep 2026 · 0 citations · 33 references
Computer Science

TL;DR

This work introduces LongSpark, a block-diffusion drafter that achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state.

Abstract

Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.

View source

Similar papers

DEdit: Iterative Draft Editing for Speculative Decoding

DEdit is introduced, a diffusion-based drafter that can not only draft by conventional parallel unmasking but also iteratively edit its draft through token-to-token predictions, and ProposalMix, a training scheme that mixes draft predictions with ground-truth tokens based on first-pass confidence during training.

Long-Xuan Yu, Bingsen Chen, Peng Shi et al. · 0 citations
#artificial intelligence Preprint Oct 2026

BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration

Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-specu...

Chen-Ce Yang, Ning-Xi Cheng, Arash Akbari et al. · 0 citations
#natural language process... Preprint Aug 2026

LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding

LiLiCorr, a Lightweight Likelihood-based model that Correlates the per-position marginal distributions a drafter already produces, is introduced, a Lightweight Likelihood-based model that Correlates the per-position marginal distributions a drafter already produces.

M. Rusanovsky, Yoav Miron, Roy Uziel et al. · 4 citations
Sep 2026

NebulaSD: Many-for-Many Speculative Decoding

NebastianSD is presented, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools.

Jun-Hao He, Hong-Yang Du · 0 citations
#artificial intelligence Preprint Oct 2026

SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as context grows and draft acceptance declines. We focus on attention-mass dilution: as softmax normalizes over more visible Keys, the mass concentra...

Jeonghoon Park, Seo-An Jo, Jongwon Lee et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.