When to Review: Spaced Repetition for Continual Pre-Training of Language Models
Spaced Repetition Training (SRT) is introduced, a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm, and preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade.
Alankar Atreya, Devesh Batra, Yoages Kumar Mantri et al.
· 1 citation