System is presented, a reference-aware codec that retrieves exact-token historical spans for prefill uplinks, reuses the reconstructed uplink state for same-round prefill downlinks, and generates boundary-specific decode references with lightweight causal predictors.
Abstract
Large language model (LLM) agents repeatedly process long, privacy-sensitive contexts, while cloud-only deployment exposes user data beyond the trusted endpoint and fully local deployment often requires costly hardware. Split inference offers a middle ground by executing the model head, tail, and tools locally and the middle layers in the cloud, but its local-cloud-local path transfers boundary hidden states at every invocation and creates a critical communication bottleneck. We present \system, a reference-aware codec that retrieves exact-token historical spans for prefill uplinks, reuses the reconstructed uplink state for same-round prefill downlinks, and generates boundary-specific decode references with lightweight causal predictors. RAC applies grouped affine alignment and calibrated residual quantization with optional prefill outliers, while sender-side wire-format reconstruction synchronizes subsequent references and offline calibration accounts for quality and packed representation costs. Across three models and nine evaluated model-link pairs, Raw-to-RAC mean time to first token (TTFT) and time per output token (TPOT) ratios are 1.24-2.72$\times$ and 1.01-2.79$\times$, while the 12 non-perplexity task-score changes range from $-0.40$ to $+2.50$ points.
This work proposes Turbo, a first-of-its-kind in-network aggregation system that accelerates long-context inference by offloading query broadcast and attention aggregation to switches and introduces a rolling forward scheme that propagates states to enable cross-stage updates.
Ying Wan, Yuchen Xu, Chuwen Zhang et al.· Conference on Applications,...· 0 citations
AceSpec, an asymmetric edge-cloud collaborative framework that employs an asymmetric communication protocol that transmits minimal main-chain indices uplink and compact sparse distributions downlink and introduces a network-aware, Lagrangian-optimized resource allocation strategy that dynamically maximizes the local cache hit rate.
Yi-Da Zhang, Zhi-Yong Gao, Shuai-Bing Yue et al.· 0 citations
Processing long, sensitive documents with machine-learning models requires efficient, privacy-preserving long-context inference. Prior private inference systems optimize or distribute encrypted Transformer attention, but its quadratic token-pair work remains the bottleneck as sequence length grows. Selective state-space models (SSMs) offer linear-time recurrence, yet direct encrypted implementation incurs linear multiplicative depth, sequence-wide state residency, or dense FHE-MPC conversion. We present Factorized Encrypted Scan-Contract (FESC), a hybrid FHE-MPC system for private long-context selective SSM inference. Its factorized scan-contract keeps input-dependent transitions compact across conversion boundaries, composes them without dense expansion, streams state chunks on demand, and contracts outputs before conversion. We demonstrate interface compatibility of the scan-contract implementation across invariant and selective SSM architectures. For our Mamba-2 instantiation, we design GPU-optimized CKKS kernels for linear computations, MPC protocols for SiLU, softplus, exponential, and RMSNorm, with approximation-aware fine-tuning. To our knowledge, FESC is the first private long-document inference system to complete native end-to-end execution at $L \geq 1{,}024$ on a single GPU. At $L = 2{,}048$, a 12-layer Mamba-base model completes inference in 77.3 minutes on one A100 GPU with a peak memory footprint of 32.7 GB, while maintaining near-plaintext accuracy on the evaluated long-document tasks.
Yufan Zhu, Chao Jin, Khin Mi Mi Aung et al.· 0 citations
On-device inference of large language models (LLMs) preserves user privacy and eliminates per-query server costs. However, weak client devices cannot run full models locally, and complete offloading negates the benefits of on-device inference. This study introduces SplitLLM, a novel zero-installation system that partitions autoregressive LLM inference between a WebGPU browser client and remote server, maintaining a key–value cache on both sides of the partition. The influence of the client device, model, split point, and network on the throughput is systematically evaluated. The results show that the optimal split point depends on the client device, whereas the per-token round trip time is invariant to the split point. This explains why adapting the split point to the network conditions yields no communication benefit, providing support for a lightweight placement rule that requires no trial offloading. The transmitted hidden state is compressed to 8 bits using static, offline-calibrated per-channel scales shared between the client and server, reducing the payload by half while keeping perplexity within 0–7% of full precision. In contrast, per-tensor quantization inflates perplexity by several orders of magnitude at intermediate split points. Under bandwidth-constrained conditions, the proposed compression speeds up decode and prefill by 1.7– $2.6\times $ and 2.4– $2.9\times $ , respectively. Overall, this work offers a promising platform for enhancing the accessibility of LLMs, especially for resource-constrained client devices.
Yuseong Lee, Yeohn-Shik Choee, Seunghyun Lee et al.· IEEE Access· 0 citations
BandwidthLLM, a communication-efficient framework that integrates three techniques: a bandwidth-aware layer placement algorithm that minimizes boundary-level transfer cost according to link bandwidth and node reliability; a lightweight activation compression scheme combining adaptive quantization with outlier-aware clipping and error feedback; and a semantic preservation check that automatically falls back to higher precision when compressed activations deviate beyond a calibrated threshold.
Zhihao Wang, Shi Shu, Yulin Wang et al.· 2026 8th International Confe...· 2 citations
This work presents a distributed inference framework that integrates speculative decoding across edge and cloud, and shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement.
D. J. Bajpai, K. Upadhyay, M. Hanawal· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
