Yu-Long Ao

SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs

Host-resident layer streaming enables full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks compete for shared host resources. Replicated transfers amplify traffic, while strong scaling can expose host work as computation windows shrink. We present SlideDP, a synchronous data-parallel runtime for sh...