SomNLP-Corpus v2: 887M+ tokens from six open sources
DatasetJun 9, 2026 · 5 min read · Updated Aug 10, 2026
SomNLP-Corpus v2: 887M+ tokens from six open sources
Our largest Somali text release yet: 1.77M clean documents, 591M words, and ~887M tokens from HPLT, CC100, mC4, OPUS, MADLAD, and MT560 — filtered through a six-stage pipeline.
SomNLP-Corpus v2 is the largest open Somali text corpus we have released: 1,774,891 clean documents, 591,321,860 words, and approximately 887 million subword tokens (~4.5 GB JSONL). It merges six public Somali-heavy distributions through a reproducible cleaning, deduplication, and language-ID pipeline.
Corpus results
Starting from 2.63M raw rows across HPLT, CC100, mC4, OPUS, MADLAD, and MT560, the pipeline merges, cleans, and LID-verifies down to 1.77M documents. Final word counts are measured on data/final/final_so.jsonl; token estimates use a ×1.5 rule-of-thumb for Somali BPE/SPM relative to whitespace-separated words.
Dataset releases, new bootcamp cohorts, research papers, and lab updates — straight to your inbox. No spam, unsubscribe anytime.
Six-stage pipeline
Each stage removes noise, duplicates, or misidentified language. We drop byte-exact duplicates after merge, apply text normalization and quality filters during cleaning, then run language identification with a Somali-specific threshold before the final export.
We prefer a smaller, verified corpus over a larger noisy dump. The largest removals happen at merge (303K rows) and LID verification (190K rows), where cross-source overlap and non-Somali content are filtered out.
Sources
v2 draws from HPLT, CC100, mC4, OPUS, MADLAD, and MT560 — the same upstream families used in multilingual pretraining research, but filtered and deduplicated specifically for Somali. License terms follow each upstream; the release bundle is CC-BY-4.0 where compatible.
How to use it
python
from datasets import load_dataset
ds = load_dataset("goobolabs/somnlp-corpus", split="train")
print(ds[0])
# Stream without a full download
ds_stream = load_dataset(
"goobolabs/somnlp-corpus", split="train", streaming=True
)
for row in ds_stream.take(10):
print(row["text"])
What's next
Public Hugging Face hosting and a full dataset card are in progress alongside model checkpoints. v3 will expand informal and conversational text — SMS, social media, and community forums.
Hugging Face release with reproducibility manifest
Held-out SomBench evaluation splits aligned to this corpus
Dialect and domain metadata on a subset of documents
Streaming API for large slices without full download
Git & GitHub: The Bootcamp That Closes Every Gap the Last Four Left Behind
Our fifth program in this series, and the first not built around data science at all. Git & GitHub closes a gap every earlier cohort surfaced: people who could write working code but weren't yet confident collaborating on it the way professional teams and open source projects do.
