Today we're releasing SomBench v0.1: the first comprehensive benchmark for Somali NLP. It covers six tasks — classification, NER, translation, summarisation, QA, and ASR — with documented splits, evaluation code, and baselines from five multilingual models.
Why we need a benchmark
Without a shared evaluation standard, researchers can't compare results. A lab in Nairobi training a Somali NLP model has no reliable way to know whether their system is better or worse than one trained in Oslo. This slows progress and makes it hard to direct resources to where they'll have the most impact.
'Somali NLP performance' is not a measurable thing. Classification accuracy on news headlines, translation BLEU on spoken dialogue, and WER on radio speech are very different problems. SomBench treats them separately — each task has its own split, metric, and leaderboard column.
The six tasks
- Text Classification — news category (6 classes), split from SomNLP-Corpus news domain
- Named Entity Recognition — 3 entity types (PER, LOC, ORG), 8K annotated sentences
- Translation — Somali↔English, 5K sentence pairs from held-out parallel corpus, BLEU metric
- Summarisation — 1K Somali news articles with human-written summaries, ROUGE evaluation
- Question Answering — 2.5K extractive QA pairs over Somali Wikipedia passages
- ASR — 20-hour held-out split of SomNLP-STT-Corpus, WER metric
Baseline results
We ran five multilingual models: mBERT, XLM-RoBERTa-base, mDeBERTa-v3-base, afroXLMR-large, and our own goobo-base. The Somali-native model leads on overall score at 52.4, despite having far fewer parameters than afroXLMR-large.
Key finding: all multilingual models underperform significantly on morphology-sensitive Somali tasks. The gap is largest on NER, where the best general model achieves only 52.1 F1 — suggesting that standard contextual representations don't capture Somali's agglutinative structure well.
How to submit your model
SomBench uses an evaluation harness that runs your model on our held-out test sets (which we don't release publicly). Install the package, generate predictions, and email us your predictions file. We score and post results within 5 business days.
# Install the evaluation harness pip install sombench-eval # Authenticate with your Hugging Face token huggingface-cli login # Run evaluation on classification task sombench-eval \ --task classification \ --model your-org/your-model \ --output predictions.jsonl # Email predictions.jsonl to sombench@goobolabs.com
Open questions
SomBench v0.1 is deliberately narrow. Important tasks we haven't covered yet: sentiment analysis, hate speech detection, machine reading comprehension, and dialect identification. v1.0 (planned for late 2026) will expand the suite and release task papers for each.
Sharafdin Yusuf
Lead Engineer & Researcher
Writing about Somali language technology, open data, and AI from the lab in Mogadishu.