AI BriefingSebastian Raschka, Qwen3 0.6B GRPO Experiment: Collapse to 0.4% at 100 Steps with KL Term, Achieves 47.4% Without It
·2026.10.28 08:30
Key point
Confirmed that removing the KL term is essential for preventing performance collapse and improving efficiency when training Qwen3 0.6B with GRPO.
Details
- Sebastian Raschka released an 11-hour video lecture and code demonstrating how to directly implement inference models using PyTorch
- On the MATH-500 benchmark, the Qwen3 0.6B base model (15.2%) improved to 47.4% after 50 steps of GRPO training with the KL term removed
- The original GRPO configuration (including the KL term) was observed to collapse to 0.4% performance after 100 steps of training
- When applying inference-time scaling (CoT, self-consistency), the base model's accuracy rose to 52.0%, but the required time increased by 85 times
- In the distillation process, teacher model data selection had a greater impact on results than the number of epochs
- Distilling with one epoch of Qwen3 235B A22B data achieved 45.0%, with training possible in approximately 30 minutes on a single H100
- The target audience is those proficient in Python and PyTorch, limited to tasks with mechanically verifiable solutions such as math problems
- The code is available on GitHub under the Apache 2.0 license, permitting commercial use
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
ProVer Framework Improves GRPO Credit Assignment for LLM Agents