Higher-Dimensional Rotary Position Embedding
HDR-RoPE is proposed, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace and significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property.
Yixing Li, Ruobing Xie, Yu-Dong Zhang et al.
· 0 citations