Self-Play Pretraining with Zero Data
Self-Play Pretraining with Zero Data is introduced, an initial proof-of-concept towards realizing a more general approach to pretraining, and whether zero-shot performance on natural data improves predictably with self-play compute is tested.