Skip to content

ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch

Sep 2026 · 2 citations · 39 references
Computer Science

TL;DR

Ufakzeka-1, a 151M-parameter (182M with embeddings) decoder-only Turkish language model pretrained from scratch on 13.5B tokens of openly licensed text and instruction-tuned for chat, is described and three findings are believed to transfer to other small-model efforts.

Abstract

We describe ufakzeka-1, a 151M-parameter (182M with embeddings) decoder-only Turkish language model pretrained from scratch on 13.5B tokens of openly licensed text and instruction-tuned for chat, at a total cost of about \$286 in cloud GPU, API and notebook time. The contribution is not the model's capability, which is what a model this size can be expected to have, but the record of building and measuring it: a Turkish byte-level tokenizer at 1.77 tokens per word, a three-stage pretraining schedule, a post-training mixture of openly licensed and generated data, and an evaluation battery of release gates, a rule-checked sweep of 5,508 conversations, judged conversations and hand tests, all with prompts held out from the training data, enforced by decontamination inside the data build and by a checked-in invariant script we run before each build. We report three findings that we believe transfer to other small-model efforts: a safety gate that had been"fixed"with training data written from its own questions read 64/64 while the honest figure was 34/64; training-seed variance was as large as the spread across every recipe we tried, so single-seed comparisons at this scale are uninformative; and data rounds repaired only what was absent from the data, while identity tracking over long context and multi-turn arithmetic did not move across any data change we tried, which we read as limits of the model size rather than gaps in the data, a reading the next, larger model will test. Weights, the data recipe, the evaluation code and the spend ledger are released under Apache-2.0.

View source

Similar papers

#natural language process... Preprint Aug 2026

Manac\'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation

Manac-a-1B is the strongest model below the 7B scale, exceeding both Tucano-1b1 and Tucano-2b4 on LAMBADA-PT with large paired margins and is competitive on commonsense completion and near chance on multiple-choice reasoning, as are all small base models.

Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fábio Porto · 0 citations
#artificial intelligence Preprint Aug 2026

Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-file C/CUDA training stack and a Devanagari-aware byte-level BPE tokenizer built for this project. On ARC-Easy and ARC-Challenge, Arkios exceeds three comparably sized op...

S. Regmi, Siddhartha Pudasaini, Chetan Phakami Pun · 0 citations
#small language model Preprint Aug 2026

MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum

MoganBert-TR, a 149M-parameter Turkish encoder foundation model trained from scratch on a language-specifically filtered corpus, together with an embedding model derived from it (MoganBert-Embed) that ranks first among student models on the MTEB(Turkish) overall average and outperforms all compared Turkish tokenizers o...

Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay · 2 citations
#natural language process... Preprint Sep 2026

CantoneseLLM v2: Reasoning in a Low-Resource Language

Evaluation across the training stages shows that chat-vector merging transfers instruction following but preserves the donor model's reasoning language, while SFT with limited Cantonese reasoning data substantially shortens or removes reasoning traces and reduces benchmark performance.

Tsz-Chung Cheng, Chung-Shing Cheng, Chaak-ming Lau et al. · 0 citations
#natural language process... Open access Sep 2026

Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model

A language model normally begins training with random word embeddings: whatever'banana'means must be learned from training corpora. I implement St. Augustine's picture of word learning, meaning by ostension, for a small masked language model (DeBERTa) trained on 10M words: before training, visually grounded tokens rece...

Lisa Bylinina · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.