LayerCheck is presented, a layer-wise adaptive checkpointing framework that selectively persists layers whose updates exceed a threshold that achieves up to 22.6× reduction in total checkpoint size and end-to-end training time compared to state-of-the-art systems, significantly lowering the cost of checkpointing.
Min-Qiu Sun, Xin Huang, Luan-Zheng Guo et al.· IEEE International Conferenc...· 1 citation
Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress can be represented by lightweight seed-and-scalar step logs, yet naive log-only recovery still incurs replay cost that grows with training...
Min-Qiu Sun, Xin Huang, Luan-Zheng Guo et al.· 1 citation
Hybrid CPU architectures have entered the mainstream desktop computing with the announcement of Intel's Alderlake architecture. Such a transition to heterogeneous CPU architecture has various performance and power implications on existing parallel workloads. In this paper we study the effects and impact of hybrid core...
Nanda Velugoti, Joseph Manzano, Nathan R. Tallent et al.· 0 citations
MEPOWER is proposed, a flexible, model-based approach to exposing compute/data movement imbalance that characterizes the fine-grained memory behavior of parallel workloads that demonstrates a reduction in EDP on a range of HPC benchmarks with minimal impact on execution time when compared to the standard OS/hardware-ma...
Nanda Velugoti, Joseph Manzano, Andrés Márquez et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
