Preprint
Oct 2026
Surprise Masking: Private Data Filtering with Model Likelihoods
Data useful for language model (LM) training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple form of perplexity filtering that uses a publicly available reference model to identify unwanted spans and suppress learning from them while...
Riya Tyagi, Sahra Ghalebikesabi, Vinnie Monaco et al.
· 0 citations