Quadratic approximations provide a common lens for studying neural network optimization, but recent evidence challenges their predictive validity. In full-batch gradient descent with learning rate (LR) $\eta$, Cohen et al. (2021) observed the Edge of Stability (EoS), where the largest Hessian eigenvalue concentrates ne...
Yu-Hang Cai, Hao-Dong Wen, Hao-Feng Huang et al.· 2 citations
The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Togeth...
2D-RoPE is introduced, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID, and suggests that viewing text in 2D can benefit language modeling.
Haodong Wen, Yiran Zhang, Yingfa Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.