Domain-Adaptive Tokenization
Extend or retrain the tokenizer on domain-specific text before fine-tuning — fewer tokens per domain concept means more content fits in the context window.
Intent & Description
🎯 Intent
A general tokenizer fragments domain-specific terms into many subword pieces, wasting context window tokens and degrading model performance on domain tasks.
📋 Context
GPT-4’s tokenizer fragments medical terms like “hypertriglyceridemia” into 7+ tokens and Python identifiers into multiple pieces. Every fragmented term means fewer real concepts fit in the context window, and the model sees arbitrary splits that the domain doesn’t have.
💡 Solution
Collect a domain corpus (medical literature, code repositories, legal documents). Train BPE or Unigram tokenizer on domain text to identify high-frequency domain tokens. Merge new domain-specific tokens into the base vocabulary (vocabulary expansion). Fine-tune the model’s embedding table for the new tokens while keeping base weights frozen. Measure token-per-word ratio before and after on representative domain text to quantify improvement.
Real-world Use Case
📌 TL;DR
When the tokenizer fragments your domain’s vocabulary into noise — extend it before fine-tuning. Fewer tokens per concept means more context, faster training, and better task performance.
Advantages
- Reduces sequence length for domain text — more content fits in the context window
- Model sees linguistically meaningful token boundaries, not arbitrary subword splits
- Improves downstream task performance on domain-specific benchmarks
Disadvantages
- Vocabulary expansion requires re-training or fine-tuning the embedding layer — not free
- New tokens have randomly initialized embeddings needing warmup steps to converge
- Larger vocabulary grows the embedding matrix and slows training