DIAMM: A Distributed Inference Acceleration Framework for Multimodal Edge Applications

Abstract

Multimodal applications such as educational assistants and medical diagnosis demand high concurrency and low latency, and are increasingly served by edge inference enhanced with Retrieval-Augmented Generation (RAG). However, heterogeneous multimodal models and fragmented edge resources couple three problems that prior work treats separately, namely compute and queue overload, unstable cold-start latency, and unreliable retrieval. Optimizing any one in isolation shifts the bottleneck to the others. To address this, we propose DIAMM (Distributed Inference Acceleration for Multimodal edge applications), which couples scheduling, cold-start, and retrieval through shared runtime signals so they co-adapt instead of competing. First, a queue management and service deployment (QMSD) algorithm adaptively prioritizes tasks and allocates service replicas under constrained edge resources. Second, a multimodal warm-up method prefetches model weights in the background to mask migration overhead, reducing cold-start latency while lowering the storage footprint. Third, a fine-tuned Question–Knowledge Assessment (QKA) module filters task-irrelevant retrieval contexts to cut prompt contamination and downstream LLM cost. We implement DIAMM on a Kubernetes-orchestrated cluster spanning x86 hosts and NVIDIA Jetson devices, and evaluate it across five multimodal tasks. Compared with baselines, DIAMM reduces average task-completion latency by 16.5% (up to 35.7%), attains a task-completion rate of up to 97.0%, and lowers average storage by about 6.3× relative to full weight replication on the x86 cluster.