Evaluating Subword Tokenization Algorithms for Kapampangan Natural Language Processing

Abstract

Effective tokenization is a foundational component of Large Language Model (LLM) development, yet it remains an open challenge for morphologically rich, lowresource languages such as Kapampangan. This paper presents a systematic evaluation of Byte-Pair Encoding (BPE) and the Unigram Language Model for Kapampangan, an under-represented Philippine language, using the SentencePiece framework. The researchers construct a monolingual corpus of 1,565,833 words drawn from Wikipedia, biblical texts, and literary sources, cleaned via LLM-assisted filtering to reduce code-switching, and train both algorithms at four vocabulary sizes (4,000; 8,000; 16,000; and 32,000). Evaluated on a 34,192-word held-out test set, both algorithms achieve zero out-of-vocabulary failures across all vocabulary sizes, demonstrating that the corpus and character-coverage configuration support robust segmentation of unseen Kapampangan text. While BPE achieves marginally more compact tokenization, the Unigram model better preserves the semantic integrity of morphologically complex words and loanwords - for example, correctly retaining teknolohiya as a single token where BPE fragments it into three linguistically uninformative pieces. Taken together with fertility and character-degradation trends across vocabulary sizes, our results support a Unigram tokenizer in the 8,000-16,000 vocabulary range as an effective and practical baseline for Kapampangan NLP.