Simple Self-Distillation Improves LLM Code Generation Performance

A simple method called Simple Self-Distillation (SSD) can dramatically improve LLM code generation performance. It doesn't require a separate verifier, teacher model, or reinforcement learning.

The method is straightforward. You collect solutions sampled from the model with specific temperature and truncation settings, then fine-tune using standard supervised learning. For Qwen3-30B-Instruct, this improved pass@1 performance on LiveCodeBench v6 from 42.4% to 55.3%, with especially large gains on harder problems.

This method works across Qwen and Llama models at 4B, 8B, and 30B sizes, and applies to both instruct and thinking versions. SSD works by resolving the precision-exploration conflict in LLM decoding. It suppresses distracting factors where precision matters, while preserving useful diversity where exploration is needed.

This suggests a new post-training direction for improving LLM code generation capabilities.