When you run a standard multilingual tokenizer on Somali text, something goes wrong. Words a Somali speaker reads as single units get shattered into fragments, inflating sequence length and wasting context and compute. We measured the damage across the full SomNLP-Corpus and trained a native BPE tokenizer that cuts Somali's token-to-word ratio from 2.69 (BERT-base) down to 1.53.
The problem
BPE tokenizers learn their subword vocabulary from statistical distributions in a training corpus. Because Somali is massively underrepresented in multilingual data, off-the-shelf tokenizers have almost no Somali-specific merges — so Somali words get decomposed with patterns learned from European languages, at character-level granularity that ignores where meaning actually sits.
The clean way to quantify this is the token-to-word ratio: how many subword tokens a tokenizer emits per whitespace-separated word. Benchmarked over all 1.67M documents of the corpus, BERT-base-uncased averages 2.69 tokens per word and XLM-RoBERTa-base 1.94, while our native Somali BPE averages 1.53. On median the gap is even wider — 1.33 native versus 2.63 for BERT-base — meaning a typical Somali document compresses better than the mean suggests.
Why Somali fragments
Two properties of Somali drive the over-segmentation. First, it is agglutinative — it builds words by stacking morphemes onto roots. Caafimaad (health) appears as caafimaadka (the health), caafimaadkiisa (his health), or caafimaadkooda (their health). Second, its Latin-script orthography and loanword conventions differ enough from English that foreign merges rarely line up with real word boundaries.
The word 'caafimaadkooda' (their health) contains: root 'caafimaad' + definite article 'ka' + 3rd-person plural possessive 'ooda'. A multilingual BPE tokenizer typically splits it at character boundaries that cross these semantic units — burning several tokens on a single meaningful word.
Our approach
Rather than bolt heuristics onto a foreign vocabulary, we trained a BPE tokenizer from scratch on Somali-only text — the final split of SomNLP-Corpus v0.2 (data/final/final_so.jsonl): 1,668,080 documents, 528,853,952 words, ~4.0 GB. The pipeline is three scripts — corpus preparation, BPE training with the Hugging Face tokenizers library, and benchmarking against baselines — with this configuration: