How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference
Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these...
C. Ekbote, Artidoro Pagnoni, Devendra Singh Sachan et al.
· 0 citations