ExpertRelay: Virtual Memory for Mixture-of-Experts Language Models Across Heterogeneous Devices

Abstract

Mixture-of-Experts (MoE) language models activate only a small subset of their parameters for each token, but they must still store all of them. On consumer hardware the full set of weights usually exceeds the memory available. This note introduces ExpertRelay, a system concept that treats the memory of one or more heterogeneous devices, together with their solid-state storage, as a single tiered virtual memory for MoE experts. A device-aware Manager profiles each device and decides where every expert lives, which experts to evict, and which to fetch ahead of need, using predictions of upcoming router decisions. This note also reports a first measurement of the simplest configuration, on a single device and without caching or prediction. On a laptop with 8.3 GB of RAM, a 14.3-billion-parameter MoE model (Qwen1.5-MoE-A2.7B, stored in int8) that does not finish loading the normal way runs at 0.702 tokens/s with a peak working set of 1,685 MB. When the operating system's own virtual memory pages the same weights from a memory-mapped file, it runs at 0.406 tokens/s with 4,431 MB. Both configurations produce identical output tokens. This note records the concept, its relation to prior work and these preliminary results. The full system and its evaluation are ongoing.