Claude Tokenizer Reproduced with ~15k Entries… Anthropic’s Vocab Reduction Strategy

Researcher Sander Land reproduced Claude’s current tokenizer, confirming it contains only about 15,000 entries. This figure contrasts with recent trends such as Qwen 3.8, which has approximately 250,000 tokens.

Gradient Bottleneck Theory

The background of this small vocabulary lies in the gradient bottleneck of the LM Head. When projecting from the model’s latent space dimension (D) to a much larger vocabulary space (V), compression occurs during backpropagation, resulting in information loss.

• For the Qwen 2.4T model, D is 8,192 and V is 250,000.
• The larger V is compared to D, the sparser the learning signal becomes, causing the model to sample only an arbitrary D-sized subset.

Anthropic’s Strategic Advantages

Anthropic reduced the vocabulary from approximately 50,000 in Claude 3 to currently about 16,000. This provides the following advantages:

• No need to project into large memory spaces, making complex optimizations like Chunked CE kernels unnecessary.
• All tokens are learned, preventing errors such as Glitch tokens.
• Rare tokens or other languages are handled via Subword tokens and UTF-8 fallback.

This approach increases the number of tokens per input text by approximately 1.2–2x, raising execution and attention costs, but avoids the learning efficiency degradation caused by gradient bottlenecks, offering a better overall trade-off.