Introducing the Somali Language Standard (SLS) – Phase 1 Complete

At Goobo Labs, we've spent years building the infrastructure required to bring the Somali language into the modern era of AI. Through our work on SomNLP Corpus v2, the Soplang programming language, and our morphological tokenizers, we've encountered one recurring, fundamental bottleneck: inconsistency.

In the world of NLP, inconsistency is the enemy of performance. Somali language datasets are riddled with conflicting spelling conventions, varying loanword adaptations, and erratic punctuation usage. Without a centralized, machine-readable source of truth for Somali orthography and grammar, every tech company, AI researcher, and linguist has had to guess — or invent their own rules.

What is the Somali Language Standard (SLS)?

The SLS is designed to act as the W3C or IETF for the Somali language. It is a strictly governed, open-source framework that centralizes the rules of the language — from the basic alphabet to complex grammar and machine-learning terminology.

Completing Phase 1: The Bedrock

You cannot build a dictionary or an LLM instruction dataset if your systems cannot agree on what constitutes a "letter." Phase 1 was entirely focused on building the constitutional framework and locking in the alphabet. We have officially published our first two standards into the Proposed state.

• SLS-0000: The Standards Process — the meta-standard, analogous to IETF's RFC 2026. Defines how all future standards are numbered, structured, versioned, and governed.
• SLS-0001: The Somali Alphabet Standard — formally defines the Somali Latin alphabet (Far Soomaali): 21 consonants, 5 vowels, collation rules for digraphs (dh, sh, kh), and vowel-length notation via doubling rather than diacritics.

The Big Technical Win: Solving Glottal Stop Tokenization

If you've ever tried to train a tokenizer or an LLM on Somali text, you are intimately familiar with the glottal stop problem. Historically, the glottal stop has been written using the ASCII apostrophe (U+0027). Standard NLP tokenizers like WordPiece or BPE classify it as punctuation (Po). When they encounter a word like su'aal (question), the tokenizer aggressively splits it into three tokens, shattering the semantic integrity of the word.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("xlm-roberta-base")

# ❌ ASCII apostrophe — classified as punctuation (Po)
tokens_before = tok.tokenize("su'aal")
print(tokens_before)  # ['su', "'", 'aal']  — 3 tokens, word is shattered

# ✅ SLS-0001: U+02BC MODIFIER LETTER APOSTROPHE (Lm)
tokens_after = tok.tokenize("suʼaal")
print(tokens_after)   # ['suʼaal']           — 1 token, semantic unit preserved

Looking Ahead: Phase 2

With Phase 1 complete and currently in its public comment period, we are immediately pivoting to Phase 2: The Orthography Standard. By building from the alphabet upwards, we are creating a mathematically sound, dependency-checked foundation for the future of the Somali language.

• SLS-0002 (Spelling Rules): Standardizing gemination, assimilation, and vowel harmony.
• SLS-0004 (Punctuation): Aligning punctuation rules with the newly locked alphabet.
• SLS-0005 (Capitalization): Defining exact case-pairing behaviors for digraphs.

We invite linguists, developers, and researchers to review the proposed standards and participate in the public comment period on our GitHub repository. Let's build the future of Somali NLP together.