Our largest Somali text release yet: 1.67M clean documents, 529M words, and ~793M tokens from HPLT, CC100, mC4, OPUS, MADLAD, and MT560 — filtered through a six-stage pipeline with deep cleaning.
SomNLP-Corpus is the largest open Somali text corpus we have released: 1,668,080 clean documents, 528,853,952 words, and approximately 793 million subword tokens (~4.0 GB JSONL). It merges six public Somali-heavy distributions through a reproducible cleaning, deep-cleaning, deduplication, and language-ID pipeline.
Corpus results
Starting from 2.63M raw rows across HPLT, CC100, mC4, OPUS, MADLAD, and MT560, the pipeline merges, cleans, LID-verifies, deep-cleans, and near-dedupes down to 1.67M documents. Final word counts are measured on data/final/final_so.jsonl; token estimates use a ×1.5 rule-of-thumb for Somali BPE/SPM relative to whitespace-separated words.
Dataset releases, new bootcamp cohorts, research papers, and lab updates — straight to your inbox. No spam, unsubscribe anytime.
Deep cleaned: 2,003,228 documents
Final: 1,668,080 documents · 528,853,952 words
Six-stage pipeline
Each stage removes noise, duplicates, or misidentified language. We drop byte-exact duplicates after merge, apply text normalization and quality filters during cleaning, run language identification with a Somali-specific threshold, deep-clean for boilerplate and segment-level LID, then near-dedup before the final export.
We prefer a smaller, verified corpus over a larger noisy dump. The largest removals happen at merge (303K rows) and near-dedup (335K rows), where cross-source overlap and near-duplicate web documents are filtered out.
Sources
The corpus draws from HPLT, CC100, mC4, OPUS, MADLAD, and MT560 — the same upstream families used in multilingual pretraining research, but filtered and deduplicated specifically for Somali. License terms follow each upstream; the release bundle is CC-BY-4.0 where compatible.
How to use it
python
from datasets import load_dataset
ds = load_dataset("goobolabs/somnlp-corpus", split="train")
print(ds[0])
# Stream without a full download
ds_stream = load_dataset(
"goobolabs/somnlp-corpus", split="train", streaming=True
)
for row in ds_stream.take(10):
print(row["text"])
What's next
Public Hugging Face hosting and a full dataset card are in progress alongside model checkpoints. v3 will expand informal and conversational text — SMS, social media, and community forums.
Hugging Face release with reproducibility manifest
Held-out SomBench evaluation splits aligned to this corpus
Dialect and domain metadata on a subset of documents
Streaming API for large slices without full download
Our Biggest Video Yet: What 12,000 Views in a Week Told Us About Somali Demand for AI Education
One week after we uploaded a 13-hour, fully Somali-language Data Science & Machine Learning course to YouTube, it passed 12,000 views — the biggest video and course we've published on the channel so far. Here's what it was, and what we're doing about what it told us.
