This work proposes Abstract Token Curriculum (ATC), a novel curriculum learning framework that elicits effective continuous intermediate representations without direct supervision or manual scratchpad design and shows ATC's effectiveness on graph reachability and arithmetic learning tasks.
Khashayar Gatmiry, Avrajit Ghosh, Parsa Mirtaheri et al.· 0 citations
It is shown that this gating mechanism causes SSMs to first learn an in-weights"memorization"solution, while delaying, or even preventing, convergence to a correct in-context learning solution, and that gating is often beneficial for improving generalization to long sequence lengths.
W. Tong, Aryo Lotfi, Emmanuel Abbé et al.· 0 citations
It is suggested that using standard CoT prompting increasingly acts as a source of distraction as models grow stronger, because standard CoT prompting also demands style adaptation, formatting compliance, and potentially undesired contextualization, which can distract models from the core reasoning task.
Denys Pushkin, Albert Q. Jiang, Aryo Lotfi et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.