FinetextPL-Edu: A High-Quality Polish Corpus for Large Language Model Pretraining via Educational Quality Filtering

Abstract

This paper presents FinetextPL-Edu, a newly developed Polish pretraining corpus filtered for educational value, constructed from the Polish slices of FineWeb2 and FinePDFs. We train a language-specific quality classifier to score the full corpus, allowing for aggressive educational filtering. We show that the removal-rate proxy underlying standard rehydration heuristics becomes misaligned after such filtering, and we evaluate alternative resampling strategies. Furthermore, we develop a Polish-optimized byte-pair encoding tokenizer that achieves the highest compression efficiency among the evaluated tokenizers. Dataset quality is empirically validated through a controlled ablation study of decoder-only language models. We find that our primary configuration, Edu-FTPL, consistently outperforms an unfiltered baseline and an external high-quality baseline across a suite of Polish-language benchmarks. The aggregate advantage is also observed in 1.8B-parameter models trained with a larger token budget, so this comparison is descriptive rather than an isolated scaling test. Finally, the native filtering configuration scores above multilingual-filter baselines on the aggregate metric, although the comparison does not isolate filtering as the causal factor.