Subword Tokenisation for Agglutinative Somali

Abstract

We study the impact of BPE, Unigram, and morphology-aware tokenization on downstream Somali NLP tasks and show that morphology-informed models reduce out-of-vocabulary rates by 33% and improve classification F1 by up to 6 points.