Skip to content

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

Sep 2026 · arXiv.org · Vol abs/2609.35440 · 0 citations · 52 references
Computer Science

TL;DR

SOLO is the first local learning method to show such memory and throughput gains in billion-parameter language-model pretraining, and becomes a practical alternative to backpropagation for large-scale pretraining.

Abstract

Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait for the gradient to pass back through every deeper layer. Conventional local learning removes this update locking by training each module to predict the target through its own readout, but has not scaled to billion-parameter pretraining. We identify these private readouts as a key weakness, since they leave each module without information from deeper modules. We propose Shared-Output LOcal learning (SOLO), which replaces them with a shared, read-only copy of the final module's readout, the only one trained on the output of the whole network. Taken from the previous step, the copy transmits information from the final module without passing gradients between modules or reintroducing update locking. SOLO approaches backpropagation on Transformers of 340M to 2B parameters pretrained on 15B tokens, staying within one point in average zero-shot accuracy with a perplexity gap that narrows with scale. Readout ablations attribute SOLO's improvement over private readouts to sharing. Without update locking, each of p pipeline stages holds activations for O(1) micro-batches instead of O(p). The freed memory permits larger micro-batches, which reach up to 1.44x the best measured throughput of pipeline backpropagation on the same partition. To our knowledge, SOLO is the first local learning method to show such memory and throughput gains in billion-parameter language-model pretraining. Local learning thus becomes a practical alternative to backpropagation for large-scale pretraining.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

Early in training, where many developmental transitions are dated, a behavioral date can reflect when the readout learns to express task information and not when the model learns it, which implies that early in training, a behavioral date can reflect when the readout learns to express task information and not when the...

Matteo He, William F. Shen, Alexandru Iacob et al. · 0 citations
#artificial intelligence Preprint Oct 2026

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, w...

Yi-Ning Li, Dong-Chen Han, Jie Fu et al. · 0 citations
#machine learning Preprint Oct 2026

Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

This work improves the two components of training that shape these fixed points: the depth prior and input injection, and learns the prior from prediction feedback, with an entropy term that keeps it broad.

Benhao Huang, Chu-Fan Shi, Jun-Lin Chen et al. · 5 citations · ⚡1

The Ups and Downs of Backprop Weights

It is argued that functional parameter identifiability may provide an organizing principle for models that systematically reuse and recombine learned functions while adapting only the components that need to change.

G. Chindemi, Benjamin Grewe · 0 citations
#artificial intelligence Preprint Oct 2026

Local Support Learning

This work proposes Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data, and presents a gate based on a Gaussian Mixture Model, whose likelihood decays rapidly away from its training data, giving it a natural ten...

Assaf Ben-Kish, Akarsh Kumar, James R. Glass et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.