Why Somali needs its own tokenizer

When you run a standard multilingual tokenizer on Somali text, something goes wrong. Words a Somali speaker reads as single units get shattered into fragments, inflating sequence length and wasting context and compute. We measured the damage across the full SomNLP-Corpus and trained a native BPE tokenizer that cuts Somali's token-to-word ratio from 2.69 (BERT-base) down to 1.53.

The problem

BPE tokenizers learn their subword vocabulary from statistical distributions in a training corpus. Because Somali is massively underrepresented in multilingual data, off-the-shelf tokenizers have almost no Somali-specific merges — so Somali words get decomposed with patterns learned from European languages, at character-level granularity that ignores where meaning actually sits.

The clean way to quantify this is the token-to-word ratio: how many subword tokens a tokenizer emits per whitespace-separated word. Benchmarked over all 1.67M documents of the corpus, BERT-base-uncased averages 2.69 tokens per word and XLM-RoBERTa-base 1.94, while our native Somali BPE averages 1.53. On median the gap is even wider — 1.33 native versus 2.63 for BERT-base — meaning a typical Somali document compresses better than the mean suggests.

Why Somali fragments

Two properties of Somali drive the over-segmentation. First, it is agglutinative — it builds words by stacking morphemes onto roots. Caafimaad (health) appears as caafimaadka (the health), caafimaadkiisa (his health), or caafimaadkooda (their health). Second, its Latin-script orthography and loanword conventions differ enough from English that foreign merges rarely line up with real word boundaries.

Our approach

Rather than bolt heuristics onto a foreign vocabulary, we trained a BPE tokenizer from scratch on Somali-only text — the final split of SomNLP-Corpus v0.2 (data/final/final_so.jsonl): 1,668,080 documents, 528,853,952 words, ~4.0 GB. The pipeline is three scripts — corpus preparation, BPE training with the Hugging Face tokenizers library, and benchmarking against baselines — with this configuration:

• 32,000-token target vocabulary, trained on Somali-only data
• Minimum merge frequency of 2 — a merge must occur at least twice to enter the vocabulary
• Whitespace pre-tokenization with NFC Unicode normalization
• Five standard control tokens: [UNK], [CLS], [SEP], [PAD], [MASK]
• Evaluated with stratified sampling (seed 42) across the entire corpus, not a small held-out slice

Results

At corpus scale the compounding is dramatic. Encoding the full release, the native tokenizer emits roughly 810M subword tokens, versus about 1.03B under XLM-RoBERTa and 1.42B under BERT-base. That's ~610M fewer tokens than BERT for the same text — fewer positions to attend over, longer effective context windows, and cheaper training and inference on every downstream Somali model.

The tokenizer

from tokenizers import Tokenizer
from transformers import AutoTokenizer

# Native Somali BPE — 32K vocab, trained on SomNLP-Corpus v0.2
som = Tokenizer.from_file("somali-bpe-tokenizer.json")
xlmr = AutoTokenizer.from_pretrained("xlm-roberta-base")

text = "Caafimaadka carruurtu waa mudnaanteenna koowaad."  # 5 words

n_som = len(som.encode(text).tokens)
n_xlmr = len(xlmr.tokenize(text))

print(n_som / 5, n_xlmr / 5)
# native BPE stays near ~1.5 tokens/word; XLM-R runs higher

# The trained tokenizer, benchmarks, and training scripts live in
# github.com/goobolabs/SomNLP-Corpus/tree/main/tokenizer