SomNLP-Corpus v2 is the largest open Somali text corpus we have released: 1,774,891 clean documents, 591,321,860 words, and approximately 887 million subword tokens (~4.5 GB JSONL). It merges six public Somali-heavy distributions through a reproducible cleaning, deduplication, and language-ID pipeline.
Corpus results
Starting from 2.63M raw rows across HPLT, CC100, mC4, OPUS, MADLAD, and MT560, the pipeline merges, cleans, and LID-verifies down to 1.77M documents. Final word counts are measured on data/final/final_so.jsonl; token estimates use a ×1.5 rule-of-thumb for Somali BPE/SPM relative to whitespace-separated words.
887M+
Subword tokens
591M
Words
1.77M
Clean documents
4.5 GB
Release size
- Downloaded (raw): 2,633,281 documents
- Merged: 2,329,800 documents
- Cleaned: 2,225,791 documents
- LID verified: 2,035,287 documents
- Final: 1,774,891 documents · 591,321,860 words
Six-stage pipeline
Each stage removes noise, duplicates, or misidentified language. We drop byte-exact duplicates after merge, apply text normalization and quality filters during cleaning, then run language identification with a Somali-specific threshold before the final export.
We prefer a smaller, verified corpus over a larger noisy dump. The largest removals happen at merge (303K rows) and LID verification (190K rows), where cross-source overlap and non-Somali content are filtered out.
Sources
v2 draws from HPLT, CC100, mC4, OPUS, MADLAD, and MT560 — the same upstream families used in multilingual pretraining research, but filtered and deduplicated specifically for Somali. License terms follow each upstream; the release bundle is CC-BY-4.0 where compatible.
How to use it
from datasets import load_dataset
ds = load_dataset("goobolabs/somnlp-corpus", split="train")
print(ds[0])
# Stream without a full download
ds_stream = load_dataset(
"goobolabs/somnlp-corpus", split="train", streaming=True
)
for row in ds_stream.take(10):
print(row["text"])What's next
Public Hugging Face hosting and a full dataset card are in progress alongside model checkpoints. v3 will expand informal and conversational text — SMS, social media, and community forums.
- Hugging Face release with reproducibility manifest
- Held-out SomBench evaluation splits aligned to this corpus
- Dialect and domain metadata on a subset of documents
- Streaming API for large slices without full download
Omar Tood
Data Scientist
Writing about Somali language technology, open data, and AI from the lab in Mogadishu.