Why Models Are Intentionally Getting Dumber

Recent LLM trends are moving toward increasing Reasoning scores while decreasing Knowledge density. While scores on reasoning benchmarks like AIME are skyrocketing, factual recall capabilities on benchmarks like SimpleQA remain low.

The Trade-off Between Knowledge and Reasoning

• The Cost of Knowledge: Factual knowledge is heavy, occupying about 2 bits of space per parameter. Storing every detail in the Weights requires the model size to grow exponentially.
• The Efficiency of Reasoning: Reasoning is much more efficient as it compresses repetitive procedures (such as problem decomposition and intermediate state tracking). It can be effectively transferred to smaller models through Synthetic data and Reinforcement Learning (RL).

Why Discard Knowledge?

• Knowledge Rot: Learned knowledge becomes outdated information immediately as time passes. Embedding knowledge into the weights is inefficient relative to its cost.
• Invariance of Procedures: Mathematical logic and problem-solving procedures do not change over time. By focusing on its role as a 'reasoning engine,' the model becomes relatively free from the issues associated with training data Cutoffs.

Conclusion: The Model is the Engine, Knowledge is the Harness The model's role is not to memorize knowledge, but to become a reasoning engine that solves problems using external tools (RAG, web search, API calls, etc.). It is more efficient and accurate to supply knowledge from the external environment at Runtime, rather than storing it in the model's internal weights.