Zipf-like Statistical Regularities in Molecular Sequence Representations for Chemical Language Models.
Analysis of rank-frequency distributions of tokens derived by byte-pair encoding (BPE) across large molecular databases shows that molecular BPE vocabularies exhibit an approximate Zipf-like rank-frequency regularity and that slope closeness provides an empirical diagnostic for comparing vocabulary sizes within the exa...