We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models. In particular, we focus on the gap between the theoretical scaling rules derived for SSMs under zero-order-hold (ZOH) discretization at inf...
Jimin Seo, Gyubok Lee, Yeonsik Jo et al.· 0 citations
This work formalizes a training rule consistent with both Masked Boundary Pause (MBP) and Masked Boundary Pause (MBP), pause tokens placed at reasoning-step boundaries with their loss masked, and demonstrates that this mode-preserving strategy extend gains to GRPO.
Jaehyeon Kim, Suhwan Kim, Nakyung Lee et al.· 0 citations
STEP-KTODER is proposed, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests and shows that execution-based labels are essential.
Idris Nechnech, Sehwan Kim, Jimin Seo et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.