Boosting Code Generation Performance with Very Simple Self-Distillation

Can we improve LLM code generation ability without a separate verifier, teacher model, or reinforcement learning (RL)? Research shows that Simple Self-Distillation (SSD), which uses only the model's own raw outputs, makes this possible.

SSD follows a very simple process: sampling solutions from the model with specific temperature and truncation settings, then retraining on them using standard supervised fine-tuning (SFT).

This approach demonstrated remarkable performance gains through experiments.

• For the Qwen3-30B-Instruct model, the pass@1 score on LiveCodeBench v6 rose significantly from 42.4% to 55.3%.
• The performance gains were especially concentrated on higher-difficulty problems.
• This effect was commonly observed across Qwen and Llama models (at 4B, 8B, and 30B scales) and both instruct and thinking variants.

The researchers traced the cause of this performance improvement to a precision-exploration conflict in the LLM decoding process. By reshaping the token distribution according to context, SSD suppresses distractor tails in regions where precision matters, while preserving useful diversity in regions that require exploration.