Cole Blondin

Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?

Mechanistic interpretability usually studies fully trained models, yet the computations that drive a behaviour can change while the model is still learning the task. On the Indirect Object Identification task, a model should continue with the name mentioned once rather than the name mentioned twice. Pythia models pass...