Characterizing Narrative Content in Web-scale LLM Pretraining Data
It is found that narrative structure is measurable at scale across extremely heterogeneous data and narrative qualities are unequally distributed across pretraining sources, topics, and formats in ways that current data curation practices neither measure nor account for.