ExpertRelay: Virtual Memory for Mixture-of-Experts Language Models Across Heterogeneous Devices
Mixture-of-Experts (MoE) language models activate only a small subset of their parameters for each token, but they must still store all of them. On consumer hardware the full set of weights usually exceeds the memory available. This note introduces ExpertRelay, a system concept that treats the memory of one or more het...