On the Design of Hyper-Connections: Simplification, Stability, and Scalability
Hyper-Connections (HC) architectures are increasingly adopted in frontier language models, yet their components remain insufficiently understood and introduce non-negligible training overhead. In this work, we systematically study the HC architecture to understand its components and identify simpler, more efficient des...
Xiang-Dong Zhang, Xiao-Han Qin, Su-Nan Zou et al.
· 0 citations