A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese
This study analyzes the predictive power of LM-derived, token-level surprisal on Mandarin Chinese reading times. We first propose the Shortest Matching Sequence (SMS), an alignment scheme that maps between the word segmentation assumed by eye-tracking corpora and the LMs'subword tokenization, as the two tokenizations o...