Wonjun Han

SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization

Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm introduces additional cross-GPU communication because the normalization factor depends on the full hidden vector. We present SwiftQK, a multi-GP...