Skip to content
Conference

Evaluating Edge and Server-Offloaded Retrieval-Augmented Generation for a Rehabilitation Voice Assistant on Raspberry Pi

Oct 2026 · 2026 20th Biennial Baltic Electronics Conference (BEC) · pp. 1-6 · 0 citations · 17 references

Abstract

Voice-based assistants offer a promising interface for rehabilitation patients with limited digital literacy, but reliable domain-specific responses require grounding in relevant knowledge rather than relying solely on language model parametric memory. Retrieval-augmented generation (RAG) provides this grounding, yet deploying RAG on resource-constrained embedded hardware introduces a non-trivial trade-off: running embedding inference on edge devices avoids network latency but strains limited CPU and RAM, while offloading to a server reduces device load but adds network round-trips. This paper evaluates both strategies for an Estonian-language rehabilitation voice assistant on Raspberry Pi 4, comparing six multilingual embedding models across 1320 test interactions. Technical pipeline performance was evaluated through per-stage latency, RAM consumption, RAG-specific energy, and retrieval quality via Recall@1/3/5; clinical safety and medical response quality were outside the study scope. The server architecture reduces end-to-end latency, measured from the end of user speech to the first audio chunk of the assistant response, by up to 40% for large unquantized models. For lightweight or quantized models, however, the reduction is less than 9%, as a fixed WebSocket overhead of 270–285 ms offsets most of the embedding savings. Speech recognition contributes a fixed 1050–1130 ms floor to all configurations regardless of deployment choice. INT8 quantization of a large embedding model reduces embedding latency by 45%, RAM by 36%, and startup time by 76% with no measurable loss in retrieval quality (Recall@5 91%), making edge deployment viable without sacrificing accuracy. Practical deployment recommendations are provided for resource-constrained, General Data Protection Regulation (GDPR)-compliant voice assistants in healthcare settings.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.