Attention Is Foundational: A Narrative Review of the Transformer Architecture from Sequence-to-Sequence to Large Language Models

Abstract

The Transformer architecture---built on attention rather than recurrence---redrew the landscape of natural language processing and became the substrate of contemporary artificial intelligence. This article presents a narrative review of the architecture's canonical line: Sutskever and colleagues' 2014 sequence-to-sequence learning, Bahdanau and colleagues' 2015 attention alignment, Vaswani and colleagues' 2017 Attention Is All You Need, Devlin and colleagues' 2019 BERT pretraining, Radford and colleagues' 2019 GPT-2, Brown and colleagues' 2020 GPT-3 and few-shot learning, Raffel and colleagues' 2020 T5 transfer, Dosovitskiy and colleagues' 2021 Vision Transformer, Bommasani and colleagues' 2021 foundation-model framing, Hoffmann and colleagues' 2022 Chinchilla scaling laws, Ouyang and colleagues' 2022 InstructGPT alignment, and Touvron and colleagues' 2023 LLaMA openness. The synthesis is organized around three themes: architecture, in which self-attention's parallel sequence processing replaced recurrence and enabled scale; scaling, in which pretraining on text plus parameter growth yielded emergent few-shot capability and then compute-optimal correction; and alignment and access, in which instruction tuning, RL from feedback, and open weights reshaped capability's deployment. It is concluded that the Transformer is machine learning's most consequential architecture to date---its attention mechanism the field's new inductive bias---and that scaling's economics and governance now define its trajectory.