COUNT Method Uses Host Memory N-gram Counts to Match 57% More Compute in 1B LMs

·2026.10.20 23:10

Key point

The method stores precomputed n-gram counts on host memory to improve validation loss and enable domain adaptation without training.

COUNT (Counting Offline Unlocks N-gram Transformers) builds n-gram embeddings without adding learned parameters by storing precomputed n-gram counts on host memory.
At a fixed compute budget, adding COUNT as an input embedding for pretraining 1B-parameter LMs improves validation loss as much as an LM with 57% more compute.
When GPU memory is limited, using COUNT to let backbone parameters fill GPU memory yields better performance than splitting memory with additional learned n-gram embeddings.
The approach allows switching to a different table of n-gram counts from a domain-specific corpus at inference time to adapt to new tasks without any training.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.