What Happens If LLMs Never See Data Above 5th Grade Level?

Modern language models (LMs) train on all data simultaneously, making it difficult to distinguish whether new skills are actually Acquired or simply Elicited. To address this, researchers built the LittleCurriculum dataset, restricted to a 5th-grade level curriculum, and introduced the LittleLearner model trained on it.

LittleLearner was created in three sizes: 0.6B, 1.3B, and 5B, with each model provided alongside an 'Unfiltered' control group sharing the same architecture and recipe.

Key research findings are as follows:

• Knowledge Limits: Model scaling, SFT+GRPO post-training, and in-context learning (ICL) all improve performance within the curriculum scope but fail to significantly improve out-of-scope performance. In other words, pre-training filters determine the model's actual capability Ceiling.
• Post-training Limits: Post-training via GRPO significantly boosts capabilities within the K-5 range but does not restore higher-order capabilities outside the scope.
• In-context Learning: Under the tested prompt conditions, ICL failed to elicit reasoning capabilities beyond the model's scope.

This experimental environment can serve as a controlled research tool for future capability discovery via RL, Continual Learning, and comparisons between human and model learning.