Somali tokenizer v2: a decode bug, a byte-level fix, and a cleaner benchmark
Our v1 Somali tokenizer couldn't reconstruct its own output — decode(encode(x)) failed on every one of 49,424 held-out documents. v2 switches to byte-level BPE, fixes it, and comes out more efficient too: 1.3528 tokens/word on a corrected, document-level benchmark.
While re-benchmarking our Somali tokenizer on a corrected, document-level held-out split, we found that v1 could not reconstruct its own input. v2 fixes the decoder with a byte-level rewrite, trains on the larger eleven-source corpus, and beats v1 on every efficiency metric while also being the first version safe to use in a generative model.
The bug in v1
v1's BPEDecoder was configured to strip an </w> end-of-word suffix that its trainer never emitted. Decoding therefore concatenated every token with no separator: 'Soomaaliya waa dal' round-tripped as 'Soomaaliyawaadal'. On our full 49,424-document held-out split, decode(encode(x)) failed on all 49,424 of them — a round-trip fidelity of 0.000. v1 also mapped out-of-alphabet characters to <unk>, which surfaced on 4 held-out documents.
For a classification or retrieval encoder, a broken decoder is invisible — you never call decode(). For a generative model it is disqualifying: every output would come back as one run-on word.
A second bug sat in the evaluation data itself: somali_raw_corpus.txt was assumed to be one document per line, but 21% of documents contain internal newlines. The old 'per-document' ratios were largely computed per paragraph for web-sourced text — including the 1.53 tokens/word figure we published for v1. That number is not comparable to anything below.
Byte-level fix
v2 switches the pre-tokenizer and decoder from whitespace/BPEDecoder to ByteLevel, and seeds the initial alphabet with all 256 bytes instead of just the characters observed in the corpus. Two properties fall out of that switch directly: decoding is exact because ByteLevel encoding is a lossless byte mapping, and <unk> becomes unreachable because every possible byte already has a token.
Dataset releases, new bootcamp cohorts, research papers, and lab updates — straight to your inbox. No spam, unsubscribe anytime.
Decoder: BPEDecoder (expected a suffix never emitted) → ByteLevel
Post-processor: none → ByteLevel(trim_offsets=True) for correct character offsets
Initial alphabet: corpus characters → all 256 bytes
Special tokens: [UNK]/[CLS]/[SEP]/[PAD]/[MASK] → <|endoftext|>, <|pad|>, <|im_start|>, <|im_end|>, plus 12 reserved slots for a decoder-LM control set
We also fixed how the evaluation split itself is built: eval_holdout.jsonl is now one JSON record per line, so documents with internal newlines stay addressable as single units, and train.py now refuses to run against a stale training file by checking a corpus fingerprint instead of just confirming the file exists.
Choosing vocabulary size
Rather than assume a vocabulary size, we swept four candidates on the same held-out split. Because BPE merges are greedy, a larger vocabulary's merge list is a superset of a smaller one's, so this isolates vocabulary size exactly.
16,384 types: 1.5210 mean tokens/word
32,000 types: 1.4058 mean tokens/word (+7.6% over 16,384)
48,000 types — shipped: 1.3528 mean tokens/word (+3.8% over 32,000)
65,536 types: 1.3195 mean tokens/word (+2.5% over 48,000, for 36% more embedding parameters)
Returns diminish steadily past 48,000. We shipped 48k as the point where v2 clears v1 on every metric at a defensible embedding cost; 65,536 stays available for deployments where context length matters more than parameter budget.
Results
All figures come from a single scoring pass over the same 49,424-document, 4,933,796-word held-out split, excluded from v2 training. v1's numbers here are a fresh, document-level measurement under the corrected protocol — not the previously published 1.53 — and v1 also saw an earlier version of these documents during training, a bias that favors v1.
1.3528
v2 mean tokens/word
1.3936
v1 mean tokens/word
1.000
v2 round-trip fidelity
0.000
v1 round-trip fidelity
v2 vs BERT-base-uncased (2.6291): 1.94× less fragmentation
v2 vs XLM-RoBERTa-base (1.8233): 1.35× less fragmentation
Corpus-level estimate at the eleven-source release (600,085,996 words): ~826M native subword tokens
The fix did not cost compression — it improved it. Freeing the merge table from spending capacity on rare corpus-specific characters, in favor of a complete byte alphabet, made v2 both correct and 2.9% more efficient than v1 on the mean ratio. The gap is widest on web-crawl sources; religious texts like Tanzil and QuranEnc stay the hardest for every tokenizer, since together they contribute well under 1% of training words.
Using the tokenizer
python
from tokenizers import Tokenizer
from transformers import PreTrainedTokenizerFast
# v2 — 48K ByteLevel BPE, exact round trips, no <unk>
tok = PreTrainedTokenizerFast(tokenizer_file="tokenizer/somali-bpe-v2.json")
text = "Caafimaadka carruurtu waa mudnaanteenna koowaad."
ids = tok.encode(text)
print(len(ids) / 5) # ~1.35 tokens/word
print(tok.decode(ids)) # reconstructs the input exactly
# Reproduce the benchmark and vocabulary sweep
# cd tokenizer && python train.py --sweep 16384,32000,48000,65536
# python benchmark.py --sweep-dir sweep
#
# github.com/goobolabs/SomNLP-Corpus/tree/main/tokenizer
Introducing the Somali Language Standard (SLS) – Phase 1 Complete
We've formally published the first two standards of the Somali Language Standard (SLS) — an open, machine-readable, CI/CD-validated framework for the Somali language, starting with the alphabet and the standards process itself.
