SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC
An SSD-native local MoE inference system that addresses challenges with an SSD I/O pipeline optimized for expert delivery, a native three-tier storage hierarchy that delivers and retains experts dynamically, and balanced CPU--GPU hybrid execution.
Fang-Zhou Liang, Yibin Shen, Jian-Min Hu et al.
· 0 citations