A comprehensive study of Tatar text classification methods: from corpus creation and Tatar2Vec embeddings to benchmark establishment

Abstract

The paper addresses the task of automatic topic classification of texts in the Tatar language, a low-resource language with rich agglutinative morphology. In the era of rapid development of large language models (LLMs), one of the key methodological challenges is the creation of open corpora, tokenizers, and annotated datasets, which are essential for both effective fine-tuning of modern neural architectures and objective evaluation of their performance. The development of such resources and the establishment of reproducible benchmarks remain an important scientific objective regardless of evolving technological paradigms. This study aims to develop an open resource base for Tatar natural language processing and to validate it experimentally in the task of topic classification. The paper presents and evaluates a set of newly developed resources, including the TatCorp 222M corpus (222 million tokens), the TatarTokenizers toolkit, the Tatar2Vec pre-trained embedding models (FastText, Word2Vec, and CBOW), and a manually annotated dataset containing 13,160 texts from ten thematic categories. The embedding models were trained on the TatCorp 222M corpus and selected from 57 configurations using a proposed composite evaluation metric. A systematic comparison of five text representation methods, including the Tatar2Vec embeddings, TF-IDF, and Bag of Words, combined with seven classification algorithms ranging from linear models and ensemble methods to multilayer perceptrons, was conducted. In addition, the influence of BPE-based subword tokenization on topic classification performance was investigated. The highest Macro $F_1$-score (0.7789) was achieved by combining pre-trained Word2Vec embeddings (without subword tokenization) with a multilayer perceptron (MLP), while a comparable Macro $F_1$-score (0.7760) and the highest ROC AUC (0.9683) were obtained using Bag of Words with XGBoost. The experiments demonstrate that BPE subword tokenization does not improve, but consistently degrades, topic classification performance for an agglutinative language (Macro $F_1$ is reduced by the amount up to 0.0234). The developed open resources and reproducible benchmarks provide a foundation for future comparative studies, including the evaluation of transformer-based models for Tatar and other low-resource Turkic languages.