The large language model (LLM) based on the Transformer architecture and its derived various applications have greatly changed people’s lives. Considering some concerns such as privacy and network conditions, deploying LLM on smart devices has gradually become a research focus. In order to reduce the huge computation and storage overhead of LLMs, many works have studied model compression technology to reduce the model computation and parameter amount, thereby reducing the inference latency. This paper analyzes the characteristics of the on-device LLM service, including small batch size and latency focus, etc. We find the inefficiency of existing model compression technologies and new optimization opportunities, i.e., allocating different layers for different input tokens based on the task QoS requirements. Then we propose CALSI, a context-aware layer skipping LLM inference system for on-device serving. In the offline profiling phase, we analyze the importance of different layers of the model to different tokens and train a lightweight gated predictor. Then, we map the latency QoS requirements of different tasks with the predictor threshold. During the online inference phase, we adaptively allocate the layers that need to be computed to the specific token and design a KV cache delayed computation management mechanism to solve the KV cache missing problem caused by layer skipping. Experiments on real devices show that CALSI can achieve up to 24.1% latency reduction. CALSI has good generalization ability on various models and datasets and is compatible with other existing complementary optimization techniques.
Pu-Han Luo, Mu Yuan, Ning-Kang Zhang et al.· IEEE Transactions on Mobile...· 0 citations
LLM agents integrated with external resources gain complex task capabilities, yet the unified natural-language context channel makes them vulnerable to injection attacks: untrusted external data may be dynamically parsed as behavior-guiding instructions during LLM inference, thereby subverting the agent's decision. Existing defenses focus on static detection or isolation of malicious content at the input/output level, remains insufficient for detecting such dynamic inducements that arise during model reasoning. We propose Attnlocate, a runtime framework for fine-grained localization of context spans that genuinely influence tool-calling decisions, i.e., behavior-guiding instructions. Attnlocate casts this localization problem as an object detection task, aiming to detect the distinctive activation traces induced by behavior-guiding instructions within the attention matrix. Specifically, we design a multi-head, multi-layer attention aggregation scheme to construct a token-level feature space tailored for object detection. Then, a 1-D U-Net equipped with an anchor-free detection head is deployed to detect these spans. Finally, based on the authority of the provider from which the detected behavior-guiding spans originate, Attnlocate dynamically adjudicates malicious invocation attempts. We evaluate Attnlocate across ten agent configurations from five LLM families, covering scenarios involving indirect prompt injection and tool poisoning. Attnlocate achieves a mean IoU of 0.743, an average AUROC of 0.956, and a 0.934 true-positive rate at 0.067 false-positive rate. It also transfers effectively across unseen models and supports authority policy adaptation without retraining.
Yichao Gao, Yumo Zhang, Yunhao Yao et al.· 1 citation
Recent advances in large vision-language models (LVLMs) have enabled long-video understanding and analysis. However, processing the large number of frames in a video incurs substantial computational overhead. Existing methods reduce LVLM inference costs by scoring frame-query relevance before inference and selecting keyframes accordingly. Nevertheless, the distribution of relevant frames varies across queries, and these methods often need to score hundreds or thousands of frames. To address this limitation, we propose CSES, a training-free semantic keyframe selector that adaptively determines the numbers of frames to score and keyframes to select. CSES estimates the prominence of the frame-query relevance profile to guide active acquisition and adapt the temporal coverage of each input. It then formulates keyframe selection as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy. Active acquisition and keyframe selection terminate based on coverage saturation. The selection objective is monotone and submodular, enabling greedy optimization with a standard approximation guarantee. Experiments with four LVLMs on two benchmarks show that our method preserves accuracy while scoring $4$-$13\times$ fewer frames and selecting $18.4\%$-$20.5\%$ fewer input keyframes than existing baselines. CSES further achieves a $3.1$-$5.4\times$ speedup in frame selection over baselines.
Jun-Yang Zhang, Puhan Luo, Chenyu Tang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
