This work extends balanced-tree recursive operators from sequence encoding to autoregressive prediction, enabling all prefix representations to be computed with logarithmic depth and linear runtime.
Abstract
Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens. Recurrent models on the other hand offer linear depth but no parallel execution. In this work, we extend balanced-tree recursive operators from sequence encoding to autoregressive prediction, enabling all prefix representations to be computed with logarithmic depth and linear runtime. Our experiments provide an initial characterization of this model class, demonstrating robust length extrapolation and performance approaching that of ALiBi-based Transformers, highlighting its potential as an alternative architecture for language modeling.
Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform...
Amr Hegazy, Amr Alanwar, Mostafa Elhoushi· 2 citations
A method for distilling sequential computation by replacing spans of input tokens with collapsed representations, computed on the fly by a lightweight merge module, allowing pretrained models to operate on compressed inputs without architectural changes or re-training.
Zi-Xuan Lan, Jessica H. Yang, Yan-Hong Li et al.· 0 citations
The Depth Control Protocol (DCP) is proposed, a diagnostic suite that disentangles three quantities of depth: truncating depth at inference time, plotting quality against retained depth fraction, and reading off the slope.
Ha Van Dau, Thanh Tung Khuat, D. Nguyen· 1 citation
The results suggest that temporal thinking offers a parameter-efficient alternative to increasing physical depth in recurrent Transformers, and that temporal thinking offers a parameter-efficient alternative to increasing physical depth in recurrent Transformers.
Ze-Yi Huang, Xuehai He, Yong Jae Lee et al.· 0 citations
Results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN, and view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout.
Ruhai Lin, Yi-Yang Guo, Ruiming Zhu et al.· 4 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
