Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
Early in training, where many developmental transitions are dated, a behavioral date can reflect when the readout learns to express task information and not when the model learns it, which implies that early in training, a behavioral date can reflect when the readout learns to express task information and not when the...