SomNLP-Corpus grows to 911M+ tokens across 13 sources
13 upstream sources, 7.35M documents, and ~912M subword tokens — plus a corrected, document-level tokenizer benchmark after we found and fixed a decode bug in v1.
The corpus has grown well past the six-source release: 13 upstream sources now feed the pipeline, producing 7,352,961 clean documents, 665,985,672 words, and approximately 911.8 million subword tokens (7.2 GB JSONL). This release also corrects the tokenizer benchmark we published earlier — the old 1.53 tokens/word figure scored lines, not documents, and is not comparable to the number below.
Corpus growth
The 2026-09-02 build is the third measured run this week: each added source moves the final count. Document growth outpaces word growth because most new sources are short, sentence-class parallel text rather than long documents.
911.8M
Subword tokens
666M
Words
7.35M
Clean documents
13
Upstream sources
Run 11 (2026-08-31, 11 sources): 6,154,594 final documents · 600M words · 826M tokens
Run 12 (2026-09-01, 12 sources): 7.24M final documents · 662M words · 906M tokens
Run 13 (2026-09-02, 13 sources): 7,352,961 final documents · 665,985,672 words · 911,824,557 tokens
Pipeline and retention
The pipeline now runs download, merge with exact dedup, clean, language ID, deep clean, and near-dedup before the final export. Deep clean is the stage we added since the six-source release: source-aware normalization, HTML and contact masking, boilerplate removal, segment-level LID, and intra-document dedup.
Dataset releases, new bootcamp cohorts, research papers, and lab updates — straight to your inbox. No spam, unsubscribe anytime.
Downloaded (raw): 17,025,862 documents
Merged: 10,790,439 documents (−6,235,423)
Cleaned: 8,104,479 documents (−2,685,960)
LID verified: 7,791,520 documents (−312,959)
Deep cleaned: 7,757,223 documents (−34,297)
Final: 7,352,961 documents (−404,262) · 665,985,672 words
The two biggest cuts are still merge (6.24M rows — cross-source overlap) and cleaning (2.69M rows — quality and length filters). Deep clean now removes another 34K rows of boilerplate and near-duplicate paragraphs the earlier stages miss.
New sources
Beyond the original six (HPLT, CC-100, mC4, OPUS, MADLAD, MT560), this release adds seven more: Wikipedia, XL-Sum, NLLB, Glot500, Somali Web Corpus, QuranEnc, and Tanzil. Document-class sources use a 25-word minimum, Somali language-ID gating, and near-dedup; sentence-class sources use a 5-word minimum and exact dedup, with language score recorded but not used as a rejection gate.
Document-class: HPLT 2.0, CC-100, mC4, MADLAD-400, Glot500, Somali Web Corpus V1, Somali Wikipedia, XL-Sum
There is no single corpus-wide license — every record carries its upstream license (CC0-1.0, CC-BY-SA-4.0, ODC-BY, CC-BY-4.0, or source-specific terms). Check a record's provenance field before redistributing.
Tokenizer, corrected
We ship two tokenizer artifacts: v1, a 32k whitespace BPE baseline we're keeping for reproducibility, and v2, the recommended 48k ByteLevel BPE. While re-running the benchmark on a deterministic, document-level held-out split (49,424 documents), we found v1's decoder was stripping a suffix its trainer never emitted — decode(encode(x)) concatenated every token, so 'Soomaaliya waa dal' came back as 'Soomaaliyawaadal' on all 49,424 held-out documents. v2's ByteLevel decoder round-trips exactly.
1.3528
Mean tokens/word (v2)
1.000
Round-trip fidelity (v2)
2.6291
BERT-base tokens/word
1.8233
XLM-RoBERTa tokens/word
The previously published 1.53 tokens/word figure is not comparable to 1.3528: it scored lines from a text file where 21% of documents contained internal newlines, so it was largely measuring paragraphs, not documents. The evaluation split is now JSONL, one document per line, and the fingerprint is checked before every training run.
How to use it
python
from datasets import load_dataset
from transformers import PreTrainedTokenizerFast
ds = load_dataset("goobolabs/somnlp-corpus", split="train")
print(ds[0])
tok = PreTrainedTokenizerFast(tokenizer_file="tokenizer/v2/tokenizer.json")
ids = tok.encode("Soomaaliya waa dal ku yaal Geeska Afrika.")
print(ids, tok.decode(ids))
What's next
Hugging Face release packaging (v0.2-clean) with a reproducibility manifest
Wikipedia and Somali web collectors, expanded beyond the current snapshot
Books, subtitles, OCR, and community contribution intake
Held-out SomBench evaluation splits aligned to this corpus
Our Biggest Video Yet: What 12,000 Views in a Week Told Us About Somali Demand for AI Education
One week after we uploaded a 13-hour, fully Somali-language Data Science & Machine Learning course to YouTube, it passed 12,000 views — the biggest video and course we've published on the channel so far. Here's what it was, and what we're doing about what it told us.
