Definition
Tokenization converts raw text into a sequence of tokens — whole words, subwords, or characters — each mapped to an ID in the model vocabulary.
LLMs do not see letters directly; they see token IDs. How text is split affects cost (API billing), context limits, and quality for morphologically rich languages like Somali.
In simple terms
Tokenization is like cutting a paragraph into puzzle pieces the model already knows. Common phrases may be one piece; rare words get split into smaller familiar fragments.
Where you see it
- OpenAI and Anthropic APIs charge per token, not per word.
- Somali-aware tokenizers reduce sequence length for Af-Soomaali text.
- Search and RAG pipelines tokenize queries before embedding.
How it works
1.Normalize text
Clean whitespace, unicode, and casing as needed.
2.Apply tokenizer rules
BPE, WordPiece, or SentencePiece split text into subword units.
3.Map to IDs
Each token maps to an integer the model embedding layer understands.
4.Decode output
Generated token IDs convert back to readable text.
Why it matters
- Token counts determine API cost and whether your prompt fits in context.
- Poor tokenization hurts low-resource languages disproportionately.
Often confused
One token equals one word.
English averages ~0.75 words per token; Somali often uses more tokens per word with multilingual tokenizers.