The three-way intersection of on-device AI inference optimization, retrieval-augmented generation, and Green AI / sustainability has not previously been drawn together into a unified system-design perspective, and this survey undertakes that cross-domain synthesis.
Edge-AI model selection is commonly driven by one isolated metric - accuracy, latency, memory, energy, or safety, even though a deployable language model must balance all five. Our work focuses on answering the question whether na- tively trained small language models (SLMs) or large language models (LLMs) compressed through post-training quantization offer the more sustainable edge- deployment trade-off. We introduce a reproducible Holistic Sustainability Score (HSS) organized around the triple bottom line: an economic pillar for capability and systems efficiency, an environmental pillar for operational GPU energy and a social pillar for harmful-prompt robustness. Five BF16 SLMs and five LLMs under different quantization approaches - BF16, INT8, NF4 4-bit, GPTQ 4-bit, and GGUF Q4 produce 30 measured configurations. Capability is assessed on five zero-shot benchmarks; efficiency uses latency, throughput, peak VRAM and energy; and safety is approximated by attack success rate on five harmful prompts. Qwen3-30B-A3B/GGUF Q4 ranks first in the combined pool (93.38), followed by Mistral-Small-24B/GGUF Q4 (92.40), while Phi-4-mini/BF16 is the highest- ranked SLM in that pool (89.49). Thus, the hypothesis that native SLMs must be the most sustainable edge choice is not supported universally; optimized quantized LLMs can win overall, while SLMs remain competitive through lower resource demand. Quantization is a systems-level choice rather than a monotonic precision- efficiency trade-off and HSS remains relative to its comparison pool and proxy definitions.
The exponential growth of the Internet of Things (IoT) has generated massive data streams traditionally processed by centralized cloud architectures, which increasingly face latency, bandwidth, and privacy limitations. Shifting artificial intelligence to resource-constrained edge nodes, known as TinyML, offers a robust decentralized alternative, though it introduces severe memory, compute, and energy bottlenecks. To map this transition, a systematic literature review was conducted following PRISMA guidelines, analyzing peer-reviewed studies published between 2021 and 2026 across major databases. The analysis identifies primary architectural paradigms and evaluates the efficacy of state-of-the-art model compression techniques, such as quantization, pruning, and knowledge distillation. Furthermore, the findings reveal that hardware–software co-design and custom neural accelerators are crucial for overcoming operational bottlenecks, while also highlighting persistent security and privacy challenges in on-device learning. Ultimately, while deploying complex models on microcontrollers is increasingly viable, achieving optimal performance demands holistic optimization strategies. This review synthesizes current research gaps and provides a strategic roadmap to guide future interdisciplinary efforts toward resilient, energy-efficient, and secure next-generation intelligent edge systems.
Marco Fiore, Francesca Lanera· Electronics· 1 citation
SmartRAG is presented, a fully on-device framework that organizes an intelligent assistant around four coordinated modules -- Perception, Memory, Focus, and Thinking -- keeping inference costs bounded and at the core of EvoNER, a continually learnable named-entity recognizer that incrementally expands its label inventory through teacher-distilled updates.
Zhihan Jiang, Meng Li, Shenghao Liu et al.· arXiv.org· 0 citations
Sixth-generation (6G) wireless networks are envisioned as AI-native systems in which semantic communication - transmitting task-relevant meaning rather than raw bits - moves beyond Shannon's classical bit-pipe model. Large language models (LLMs) dominate semantic encoding but are unsuitable for 6G user equipment and IoT devices, given prohibitive memory, energy, and latency costs. Tiny language models (TinyLMs) - compressed via TinyML techniques into kilobyte-to-megabyte memory and milliwatt power budgets - are the missing bridge between LLM-level semantic encoding and 6G edge hardware, yet no prior work systematically maps TinyML techniques onto semantic communication architectures for this purpose. This survey closes that gap through a two-axis taxonomy connecting six compression families (quantization, pruning, knowledge distillation, low-rank adaptation, neural architecture search, hybrid pipelines) to five semantic communication architectures (end-to-end joint source-channel coding, split learning, federated learning, knowledge-graph-assisted, and multi-task/cross-modal communication), synthesized with a quantitative meta-analysis of the model-size-versus-semantic-fidelity Pareto frontier. Representative results include a CNN-Transformer encoder achieving 22 dB PSNR at 33.33% semantic-representation size reduction; a symbolic protocol machine reducing a neural MAC protocol from 4.55 MB to 1 KB (99.98% smaller) with zero performance loss; federated bidirectional knowledge distillation converging under joint model-and-data heterogeneity where FedAvg-style averaging underperforms; and knowledge-graph-assisted probability graphs cutting transmission energy by 65%. The survey identifies nine open research challenges for TinyLM-enabled 6G semantic communication, including two not previously articulated in the literature.
Srikanth Kamath, Arnav Mathur, Joslyn Sajan George et al.· 0 citations
This work surveys dozens of recent works that report compression results on real hardware and extracts practical deployment guidelines from them, and deploys compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation.
Subhransu Das, Jiaming Cheng, Arnav Kumar et al.· 0 citations
The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment and hardware-aware design. In this work, we introduce Hybrid Modeling for Energy and Latency of LLMs (HYMELL), a hybrid three-level framework for estimating LLM inference latency and energy by combining analytical modeling with machine learning (ML). HYMELL models LLM execution through a three-level hierarchy: analytical estimation of primitive operations, ML prediction of higher-level components, and an end-to-end model that captures system-level overheads across both prefill and decode phases. The framework supports diverse architectures, including dense and mixture-of-experts (MoE) feed-forward networks (FFNs), as well as multi-head attention (MHA) and grouped-query attention (GQA) mechanisms. Evaluated on an NVIDIA H100 graphics processing unit (GPU), HYMELL achieves high predictive accuracy; notably, for LLaMA 3 8B, it attains less than 5% error for both prefill and decode phases. By predicting execution costs directly from architectural parameters, it enables fast, hardware-free design space exploration and energy-efficient optimization.
Saeid Shokoufa, Mohammad Erfan Sadeghi, M. Kamal et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
