Qiaoling Chen

SPPO: Making Million-Token LLM Training Practical on Modest GPU Clusters

Adaptive Sequence Pipeline Parallel Offloading (SPPO) is proposed, a novel framework that optimizes memory and computational resource efficiency for long-sequence LLM training and develops an adaptive pipeline scheduling approach with a heuristic solver and multiplexed sequence partitioning to improve computational resource efficiency.