As large language models (LLMs) grow more powerful, their scale makes them costly to serve. Depth pruning offers a practical route to efficiency by directly reducing memory and latency without hardware-specific support. However, we observe that transformer layers exhibit markedly different task-dependent layer contribu...
Guan-Chen Li, Yi-Xing Xu, Xuan-Wu Yin et al.· 0 citations
CompassPlay is introduced, a self-play method that rewards the proposer through gradient alignment that draws on a first-order approximation to learning progress and scores each eligible task without additional solver training.
Sophia Xiao Pu, Xi-Meng Sun, Jiang Liu et al.· 0 citations
Instella-MoE is introduced, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs, establishing a strong, fully open foundation for efficient, high-performing MoE models and r...
Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.