SmartRAG is presented, a fully on-device framework that organizes an intelligent assistant around four coordinated modules -- Perception, Memory, Focus, and Thinking -- keeping inference costs bounded and at the core of EvoNER, a continually learnable named-entity recognizer that incrementally expands its label inventory through teacher-distilled updates.
Abstract
Deploying large language models (LLMs) as personal assistants on mobile devices demands privacy, low latency, and offline availability, yet the computational cost of giant models clashes with strict edge-hardware budgets. We argue that this tension cannot be resolved by model compression alone; it requires decomposing on-device intelligence into complementary functional roles. We present SmartRAG, a fully on-device framework that organizes an intelligent assistant around four coordinated modules -- Perception, Memory, Focus, and Thinking. At the core of SmartRAG is EvoNER, a continually learnable named-entity recognizer that incrementally expands its label inventory through teacher-distilled updates, enabling the system to absorb previously unseen entity types without retraining the backbone LLM. Extracted knowledge is stored in MRGraph, a three-layer provenance-preserving knowledge graph, and retrieved at query time through a hybrid pipeline combining graph traversal, lexical matching, and dense semantic search. The on-device LLM is invoked only for high-value semantic operations -- labeling, planning, and answer synthesis -- keeping inference costs bounded. Experiments on four QA benchmarks (TriviaQA, Natural Questions, HotpotQA, MultiHopQA) show that SmartRAG with a quantized 1.7B-parameter backbone achieves multi-hop reasoning performance competitive with models up to 18$\times$ larger, while running entirely on commodity smartphones within practical memory and latency envelopes.
Integrating Knowledge Graphs (KGs) into Retrieval-Augmented Generation (RAG) can substantially improve LLM performance on complex question answering (QA) by reducing hallucinations and supplying structured context. However, building high-quality KGs over large corpora for edge scenarios is challenging: cloud-based processing introduces latency and dependency on remote services, while exhaustive on-device construction with LLMs is often computationally infeasible under limited hardware budgets. We observe that traditional non-LLM methods can efficiently capture explicit knowledge, and that real-world queries typically touch only a small, highly concentrated portion of the graph. As a result, static and exhaustive KG construction is redundant and inefficient. We propose Edge-AdaptiveKG, a resource-aware framework that combines an offline Seed KG (S-KG) with an online Query-driven KG (Q-KG). Lightweight non-LLM methods build the S-KG, while the LLM is invoked on demand during question answering to incrementally expand the Q-KG only when complex relations are needed. Experiments show that Edge-AdaptiveKG reduces computational overhead and inference latency, enabling KG-enhanced RAG on resource-constrained devices while maintaining competitive QA accuracy.
Yuyu Du, Juxin Niu, Chun Jason Xue et al.· IEEE International Conferenc...· 0 citations
This work presents SemanticXR, the first device-cloud system for real-time, open-vocabulary semantic mapping and querying under XR power, bandwidth, and memory constraints, and key insight is to elevate semantically identifiable objects to first-class units of system design, governing how the system communicates, executes, and manages memory across the device and the server.
Rahul Singh, Devdeep Ray, Connor Smith et al.· 0 citations
Personalized Query prediction maps implicit behavioral signals---clicks, favorites, purchases, and post-purchase exploration---to explicit retrieval intent. On-device deployment makes this task particularly challenging: behavioral trajectories are noisy and multi-scale, multiple Queries may be valid for a single trajectory, and a uniform reasoning policy either expends unnecessary computation on simple instances or allocates insufficient capacity to complex ones. We introduce RecGPT-Mobile-V2, an end-to-end framework that treats intent quality and execution efficiency as coupled objectives within a staged design. The framework transforms heterogeneous interactions into an evidence-preserving trajectory, establishes a recommendation-native foundation through domain adaptation and supervised alignment, and applies reasoning-cost optimization only after grouped rollouts meet grounding and utility criteria. The resulting teacher is distilled into a compact student deployed with low-bit execution, structured compression, and budget-aware device--cloud routing. In an aligned CoT ablation, an evidence-focused short rationale increases ROUGE-L from 0.228 to 0.315 and Jaccard from 0.174 to 0.248, while slightly outperforming the full five-stage rationale. In the controlled RL comparison, the complete reward formulation improves Query quality from 73.2% under quality-only RL to 78.6%, lowers the hard-failure rate from 3.6% to 1.6%, and reduces the median CoT length from 62 to 14 tokens. Online retrieval analysis further indicates that the Query recall channel retrieves inventory complementary to that surfaced by established recall channels. Collectively, these findings support sufficiency-oriented rather than uniformly short reasoning: retain decision-relevant evidence and allocate additional computation only when it is likely to improve the predicted Query.
Lingqin Zhang, Bin Zhang, Wei-Peng Huang et al.· 0 citations
Results show that DEFRAG narrows the SLM-LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services, demonstrating the potential of DEFRAG for democratized LLM services at the edge.
Jiaxing Li, Hengzhi Wang, Feng Wang et al.· IEEE Transactions on Mobile...· 0 citations
The three-way intersection of on-device AI inference optimization, retrieval-augmented generation, and Green AI / sustainability has not previously been drawn together into a unified system-design perspective, and this survey undertakes that cross-domain synthesis.
Zhiyuan Cheng, Longying Lai, Yue Liu et al.· 6 citations
This work presents a distributed inference framework that integrates speculative decoding across edge and cloud, and shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement.
D. J. Bajpai, K. Upadhyay, M. Hanawal· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
