When Can LLMs Learn Reasoning Under Weak Supervision

RLVR generalization was governed by saturation dynamics rather than data quantity. A long pre-saturation phase, where training reward steadily rises, leads to transferable reasoning, but once the model enters post-saturation, where reward flattens out, the gains from further training sharply diminish.

• Scarce data: Even at N=8, Qwen-Math-1.5B kept training for 342 steps, with MATH-500 rising from 35% to 67%. Qwen-1.5B saturated at 172 steps, and Llama-3B-Instruct saturated at 60 steps.

• The same pattern repeated across Math, Science, and Graph, and on Graph, where pretraining exposure is low, even Qwen-Math saturated faster.

• Noisy rewards: For Llama-3B-Instruct on Math, as the corruption rate γ went from 0.1 to 0.9, MATH-500 dropped from about 51% to 42%. Qwen-Math-7B on Graph held up under low corruption but weakened sharply once γ reached 0.5 or higher.

• Proxy rewards: majority vote and self-certainty were fragile. Qwen-3B briefly improved under majority vote before collapsing after about 500 steps, and Llama-3B-Instruct reward-hacked majority vote to 1.0, causing MATH-500 to collapse from 45% to 4%. self-certainty collapsed for both models.

The cause of failure was not a lack of output diversity but unfaithful reasoning. Llama showed higher diversity than Qwen, but it produced chains-of-thought that did not logically support the final answer, effectively memorizing correct answers.

The solution was continual pre-training (CPT) and Thinking SFT. Performing CPT with domain-specific data (e.g., 52B math tokens), first learning explicit reasoning from 43.5K reasoning traces, and then adding RL (GRPO) raised faithfulness and lengthened pre-saturation, restoring generalization across all three weak supervision settings.