Qwen4Exp: N-gram Architecture Analysis

The Qwen4Exp architecture relies on N-gram tables rather than Mixture of Experts (MoE) for parameter offloading. While MoE focuses on arithmetic operations where a router inspects hidden states to select necessary feed-forward blocks, N-grams perform memory operations by generating addresses from token hashes to store vectors for short phrases.

N-gram tables have addresses as soon as tokens exist, collecting only KB-sized rows without multiplying the entire table and folding them into the stream. This allows the 176B model to consist of 125B in RAM and 51B in SSD, with only about 6B activated per token, enabling it to utilize 176B trained parameters at speeds similar to a 125B-A6B model.

Technical Limitations and Optimization

• Offloading Limit: Allocating approximately 20–25% of total parameters to N-gram tables is optimal; exceeding this eliminates the benefits.
• Storage: Storing N-gram tables on SSD instead of RAM provides the greatest advantage.
• Role Division: MoE handles reasoning, while N-grams handle recalling. Since recalling alone degrades quality, combining both approaches is essential.