Today we're releasing SomBench v0.1: the first comprehensive benchmark for Somali NLP. It covers six tasks — classification, NER, translation, summarisation, QA, and ASR — with documented splits, evaluation code, and baselines from five multilingual models.
Why we need a benchmark
Without a shared evaluation standard, researchers can't compare results. A lab in Nairobi training a Somali NLP model has no reliable way to know whether their system is better or worse than one trained in Oslo. This slows progress and makes it hard to direct resources to where they'll have the most impact.
'Somali NLP performance' is not a measurable thing. Classification accuracy on news headlines, translation BLEU on spoken dialogue, and WER on radio speech are very different problems. SomBench treats them separately — each task has its own split, metric, and leaderboard column.
The six tasks
Text Classification — news category (6 classes), split from SomNLP-Corpus news domain
Sii daynta xogta, kooxo bootcamp cusub, waraaqo cilmi-baaris, iyo cusboonaysiinta shaybaarka — toos ugu socda sanduuqaaga. Spam ma jiro, waad ka bixi kartaa markasta.
We ran five multilingual models: mBERT, XLM-RoBERTa-base, mDeBERTa-v3-base, afroXLMR-large, and our own goobo-base. The Somali-native model leads on overall score at 52.4, despite having far fewer parameters than afroXLMR-large.
Key finding: all multilingual models underperform significantly on morphology-sensitive Somali tasks. The gap is largest on NER, where the best general model achieves only 52.1 F1 — suggesting that standard contextual representations don't capture Somali's agglutinative structure well.
How to submit your model
SomBench uses an evaluation harness that runs your model on our held-out test sets (which we don't release publicly). Install the package, generate predictions, and email us your predictions file. We score and post results within 5 business days.
bash
# Install the evaluation harness
pip install sombench-eval
# Authenticate with your Hugging Face token
huggingface-cli login
# Run evaluation on classification task
sombench-eval \
--task classification \
--model your-org/your-model \
--output predictions.jsonl
# Email predictions.jsonl to sombench@goobolabs.com
Open questions
SomBench v0.1 is deliberately narrow. Important tasks we haven't covered yet: sentiment analysis, hate speech detection, machine reading comprehension, and dialect identification. v1.0 (planned for late 2026) will expand the suite and release task papers for each.
