Looped language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization. We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test...
Benhao Huang, Chu-Fan Shi, Jun-Lin Chen et al.· 4 citations
Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This proxy can be misleading for softmax, where the RMS shift in activations does not faithfully measure the change in the o...
Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno et al.· 0 citations
Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect a shared deficit in visual transition reasoning, and test whether this capability can serve as a shared training primitive, one that different models can learn from diff...
Zhe-Yu Fan, Yue Zhang, Ming-Kai Deng et al.· 0 citations
Every recurrence of a looped language model adds cost in training, decoding, prefill, and reinforcement learning (RL). The closer recurrent states get to fixed points, the less the path to them matters. This enables truncated backpropagation in training; terminal key-value (KV) sharing for decoding with almost no loss...
Benhao Huang, Chu-Fan Shi, Jun-Lin Chen et al.· 1 citation
Fluorescence microscopy reveals where proteins localize, but only a limited number of proteins can be imaged in the same cell; generating these images from amino-acid sequence and the cell's morphological context enables in silico localization of unimaged proteins. The two conditions, however, play asymmetric roles: mo...
Sazan Mahbub, Caleb N. Ellington, E. Xing· 0 citations
Muon motivates designing optimizer geometry around the function of each parameter block and uses the spectral norm for hidden linear layers. For the language-model head, the spectral norm is not a faithful measure of functional change. Softmax removes shared logit shifts, whereas the spectral norm can assign arbitraril...
Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno et al.· 0 citations
WM-VLM is introduced, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states and suggests that internal world models offer a promising path toward VLMs that reason in both language and visual space.
Yuheng Zha, Yi-Lei Wang, Qiyue Gao et al.· 0 citations
GProj (gauge projection), which restores the zero sum after the cast with two rank-one corrections per row, and cuts the remaining median query/key gradient errors from 219%/13% to 0.34%/0.37%, on par with FP32 attention, for 4.7% more time per training step.
Jun-Lin Chen, Daize Dong, Huan-Wei Di et al.· 0 citations
K2-V2 is introduced, a 360-open LLM built from scratch as a superior base for reasoning adaptation, in addition to functions such as conversation and knowledge retrieval from general LLMs, to empower the community with a capable, reasoning-centric foundation.
K2 Team Zhengzhong Liu, Liping Tang, Ling-Hao Jin et al.· 0 citations
Chronosphere is introduced, a spatio-temporal neural field that learns representations of climate that matches or leads state-of-the-art location encoders across spatial and temporal tasks, with the largest gains under spatial and temporal transfer.
Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video...
Zu-Xing Lu, Hong-Jia Zhai, Guan-Zhi Wang et al.· 4 citations
This work proposes a generational genetic algorithm to coordinate specialized large language model agents that integrate mechanistic arguments, reconsider assumptions, and assess evidence and testability that advances a vision of autonomous science in which AI research teams achieve a capacity for discovery beyond that...
Jie-Yuan Liu, Meng-Zhou Hu, Jefferson Chen et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
