AI BriefingResearch Shows Frozen 16 MB Vocabulary Outperforms Learned Embeddings in Small Language Models
·2026.10.21 22:50
Key point
A frozen 16 MB vocabulary table matches or exceeds learned embeddings on key benchmarks while reducing model size to 46 MB and boosting CPU inference speed by five to six times.
Details
- Replacing the standard learned embedding table with a frozen 16 MB vocabulary of 512-bit codes reduces trainable parameters by 2.6 times while maintaining or improving performance on LAMBADA, HellaSwag, and PIQA.
- The frozen table achieves a 0.20 nat validation loss penalty compared to the learned table but wins by 1.7 to 2.3 nats on rare words and handles 10,000 new words with only a 3% cost, whereas the learned table collapses by 43%.
- When combined with a ternary BitNet b1.58 body, the entire model compresses to a 46 MB file, generating 615 to 740 tokens per second on a laptop CPU with 62 MB of memory.
- This CPU inference speed is five to six times faster than the float model's 8-bit GGUF version and uses one-seventh of the memory, with all weights, data, and evaluation answers released on GitHub and Hugging Face.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Hybrid Reasoning Agents achieve 35-55% token reduction and up to 2.6x faster inferenceResearchers Introduce Context Language Models for Native Context Management